From 0b2891e33cbc8aac37f34f591224d91e1d33ef73 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Wed, 8 Apr 2026 13:37:17 -0400 Subject: [PATCH 01/13] REFACTOR: Core out-of-core (OOC) architecture rewrite for simplnx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the chunk-based DataStore API with a plugin-driven hook architecture that cleanly separates OOC policy (in the SimplnxOoc plugin) from mechanism (in the core library). The old API required every caller to understand chunk geometry; the new design hides OOC details behind bulk I/O primitives and plugin-registered callbacks. --- AbstractDataStore / IDataStore API --- Remove the entire chunk API from AbstractDataStore and IDataStore: loadChunk, getNumberOfChunks, getChunkLowerBounds, getChunkUpperBounds, getChunkShape, getChunkSize, getChunkTupleShape, getChunkExtents, and convertChunkToDataStore. Replace with two bulk I/O primitives: copyIntoBuffer(startIndex, span) and copyFromBuffer(startIndex, span), implemented in DataStore (std::copy on raw memory) and EmptyDataStore (throws). This shifts the abstraction from "load a chunk, then index into it" to "copy a contiguous range into a caller- owned buffer," which works identically for in-core and OOC stores. Simplify StoreType to three values (InMemory, OutOfCore, Empty) by removing EmptyOutOfCore. IsOutOfCore() now checks StoreType instead of testing getChunkShape().has_value(). Add getRecoveryMetadata() virtual to IDataStore for crash-recovery attribute persistence. --- Plugin Hook System (DataIOCollection / IDataIOManager) --- Add three plugin-registered callback hooks to DataIOCollection: FormatResolverFnc: Decides storage format for a given array based on type, shape, and size. Called from DataStoreUtilities::CreateDataStore and CreateListStore. Replaces the removed checkStoreDataFormat() and TryForceLargeDataFormatFromPrefs — format decisions now live entirely in the plugin, with core only calling resolveFormat() when no format is already set. BackfillHandlerFnc: Post-import callback that lets the plugin finalize placeholder stores after all HDF5 objects are read. Called from ImportH5ObjectPathsAction after importing all paths. Replaces the removed backfillReadOnlyOocStores core implementation. WriteArrayOverrideFnc: Intercepts HDF5 writes during recovery file creation, allowing the plugin to write lightweight placeholder datasets instead of full array data. Activated via RAII WriteArrayOverrideGuard, wired into DataStructureWriter. Add factory registration on IDataIOManager for ListStoreRefCreateFnc, StringStoreCreateFnc, and FinalizeStoresFnc, with delegating creation methods on DataIOCollection. Guard against reserved format name "Simplnx-Default-In-Memory" during IO manager registration. --- EmptyStringStore Placeholder --- Add EmptyStringStore, a placeholder class for OOC string array import that stores only tuple shape metadata. All data access methods throw std::runtime_error. isPlaceholder() returns true (vs false for StringStore). StringArrayIO creates EmptyStringStore in OOC mode instead of allocating numValues empty strings. --- HDF5 I/O --- DataStoreIO::ReadDataStore gains two interception paths before the normal in-core load: (1) recovery file detection via OocBackingFilePath HDF5 attributes, creating a read-only reference store pointing at the backing file; (2) OOC format resolution via resolveFormat(), creating a read-only reference store directly from the source .dream3d file with no temp copy. DataArrayIO::writeData always calls WriteDataStore directly — OOC stores materialize their data through the plugin's writeHdf5() method; recovery writes use WriteArrayOverrideFnc. NeighborListIO gains OOC interception: computes total neighbor count, calls resolveFormat(), and creates a read-only ref list store when an OOC format is available. Legacy NeighborList reading passes a preflight flag through the entire call chain (readLegacyNeighborList -> createLegacyNeighborList -> ReadHdf5Data) so legacy .dream3d imports create EmptyListStore placeholders instead of eagerly loading per- element via setList(). DataStructureWriter checks WriteArrayOverrideFnc before normal writes, giving the registered plugin callback first chance to handle each data object. Add explicit template instantiations for DatasetIO::createEmptyDataset and DatasetIO::writeSpanHyperslab for all numeric types plus bool. These are needed by the SimplnxOoc plugin's AbstractOocStore::writeHdf5(), which cannot use writeSpan() because the full array is not in memory. Instead it creates an empty dataset, then fills it region-by-region via hyperslab writes as it streams data from the backing file. --- Preferences --- Add unified oocMemoryBudgetBytes preference (default 8 GB) that the ChunkCache, visualization, and stride cache all use. Add k_InMemoryFormat sentinel constant for explicit in-core format choice. Add migration logic to erase legacy empty-string and "In-Memory" preference values. checkUseOoc() now tests against k_InMemoryFormat. setLargeDataFormat("") removes the key so plugin defaults take effect. --- Algorithm Infrastructure --- AlgorithmDispatch: Add ForceInCoreAlgorithm/ForceOocAlgorithm global flags with RAII guards. Add DispatchAlgorithm template that selects Direct (in-core) vs Scanline (OOC) algorithm variant based on store types and force flags. Add SIMPLNX_TEST_ALGORITHM_PATH CMake option (0=both, 1=OOC-only, 2=InCore-only) for dual-dispatch test control. IParallelAlgorithm: Remove blanket TBB disabling for OOC data — OOC stores are now thread-safe via ChunkCache + HDF5 global mutex. CheckStoresInMemory/CheckArraysInMemory use StoreType instead of getDataFormat(). VtkUtilities: Rewrite binary write path to read into 4096-element buffers via copyIntoBuffer, byte-swap in the buffer, and fwrite — replacing direct DataStore data() pointer access. --- Filter Algorithm Updates --- FillBadData: Rewrite phaseOneCCL and phaseThreeRelabeling to use Z-slab buffered I/O via copyIntoBuffer/copyFromBuffer instead of the removed chunk API (loadChunk, getChunkLowerBounds, etc.). operator()() scans feature counts in 64K-element chunks via copyIntoBuffer. QuickSurfaceMesh: Remove getChunkShape() call in generateTripleLines() that set ParallelData3DAlgorithm chunk size, as the chunk API no longer exists on AbstractDataStore. --- File Import --- ImportH5ObjectPathsAction: Add deferred-load pattern. When a backfill handler is registered, pass preflight=true to create placeholder stores during import, then call runBackfillHandler() after all paths are imported to let the plugin finalize. Dream3dIO: Add WriteRecoveryFile() that wraps WriteFile with WriteArrayOverrideGuard. --- Utility Changes --- DataStoreUtilities: Remove TryForceLargeDataFormatFromPrefs entirely. CreateDataStore and CreateListStore call resolveFormat() on the IO collection. ArrayCreationUtilities: check k_InMemoryFormat sentinel before skipping memory checks. ITKArrayHelper/ITKTestBase: OOC checks use getStoreType() instead of getDataFormat().empty(). IsArrayInMemory simplified from a 40-line DataType switch to a single StoreType check. ArraySelectionParameter: Remove EmptyOutOfCore handling; simplify to just StoreType::Empty. --- Tests --- Add EmptyStringStore tests (6 cases: metadata, zero tuples, throwing access, deep copy placeholder preservation, resize, isPlaceholder). Add DataIOCollection hooks tests (format resolver, backfill handler). Add IOFormat tests (7 cases: InMemory sentinel, empty format, resolveFormat with/without plugin). Add IParallelAlgorithm OOC tests (8 cases with MockOocDataStore: TBB enablement for in-memory, OOC, and mixed arrays/stores). Remove the "Target DataStructure Size" test from IOFormat.cpp — it was a tautology that re-implemented the same arithmetic as updateMemoryDefaults() without testing any edge case or behavior. Fix RodriguesConvertorTest exemplar data: add missing expected values for the 4th tuple (indices 12-15). The old CompareDataArrays broke on the first floating-point mismatch regardless of magnitude, masking this incomplete exemplar. The new chunked comparison correctly continues past epsilon-close differences, exposing the missing data. Signed-off-by: Joey Kleingers --- CMakeLists.txt | 11 + cmake/Plugin.cmake | 1 + .../Common/ITKArrayHelper.hpp | 4 +- .../ITKImageProcessing/test/ITKTestBase.cpp | 44 +-- .../test/RodriguesConvertorTest.cpp | 4 + .../Algorithms/ComputeArrayStatistics.cpp | 3 +- .../Filters/Algorithms/FillBadData.cpp | 254 ++++++--------- .../Filters/Algorithms/QuickSurfaceMesh.cpp | 5 - .../src/SimplnxCore/utils/VtkUtilities.hpp | 39 ++- src/simplnx/Core/Preferences.cpp | 68 +++- src/simplnx/Core/Preferences.hpp | 43 ++- .../DataStructure/AbstractDataStore.hpp | 147 ++------- .../DataStructure/AbstractStringStore.hpp | 43 ++- src/simplnx/DataStructure/DataStore.hpp | 102 ++---- src/simplnx/DataStructure/EmptyDataStore.hpp | 61 +--- .../DataStructure/EmptyStringStore.hpp | 207 ++++++++++++ src/simplnx/DataStructure/IDataStore.hpp | 32 +- .../IO/Generic/DataIOCollection.cpp | 170 +++++++++- .../IO/Generic/DataIOCollection.hpp | 224 ++++++++++++- .../IO/Generic/IDataIOManager.cpp | 82 +++++ .../IO/Generic/IDataIOManager.hpp | 103 ++++++ .../DataStructure/IO/HDF5/DataStoreIO.hpp | 146 ++++++++- .../IO/HDF5/DataStructureWriter.cpp | 35 ++- .../DataStructure/IO/HDF5/NeighborListIO.hpp | 73 ++++- .../DataStructure/IO/HDF5/StringArrayIO.cpp | 23 +- src/simplnx/DataStructure/StringStore.hpp | 32 +- .../Actions/ImportH5ObjectPathsAction.cpp | 29 +- .../Parameters/ArraySelectionParameter.cpp | 6 +- src/simplnx/Utilities/AlgorithmDispatch.hpp | 114 ++++++- .../Utilities/ArrayCreationUtilities.cpp | 16 +- src/simplnx/Utilities/DataStoreUtilities.cpp | 10 - src/simplnx/Utilities/DataStoreUtilities.hpp | 19 +- src/simplnx/Utilities/IParallelAlgorithm.cpp | 15 +- .../Utilities/Parsing/DREAM3D/Dream3dIO.cpp | 40 ++- .../Utilities/Parsing/DREAM3D/Dream3dIO.hpp | 13 + .../Utilities/Parsing/HDF5/H5DataStore.hpp | 4 +- .../Utilities/Parsing/HDF5/IO/DatasetIO.cpp | 108 +++++++ .../Utilities/Parsing/HDF5/IO/DatasetIO.hpp | 39 +++ test/CMakeLists.txt | 3 + test/DataIOCollectionHooksTest.cpp | 44 +++ test/EmptyStringStoreTest.cpp | 60 ++++ test/IOFormat.cpp | 123 +++++++- test/IParallelAlgorithmTest.cpp | 296 ++++++++++++++++++ .../simplnx/UnitTest/UnitTestCommon.hpp | 73 +++-- 44 files changed, 2348 insertions(+), 620 deletions(-) create mode 100644 src/simplnx/DataStructure/EmptyStringStore.hpp create mode 100644 test/DataIOCollectionHooksTest.cpp create mode 100644 test/EmptyStringStoreTest.cpp create mode 100644 test/IParallelAlgorithmTest.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 8841931bfe..be6b83f810 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -53,6 +53,14 @@ option(SIMPLNX_DOWNLOAD_TEST_FILES "Download the test files" ON) # ------------------------------------------------------------------------------ option(SIMPLNX_WRITE_TEST_OUTPUT "Write unit test output files" OFF) +# ------------------------------------------------------------------------------ +# Controls which algorithm paths are exercised by dual-dispatch unit tests. +# 0 (Both) - tests run with forceOoc=false AND forceOoc=true (default) +# 1 (OocOnly) - tests run with forceOoc=true only (use for OOC builds) +# 2 (InCoreOnly) - tests run with forceOoc=false only (quick validation) +# ------------------------------------------------------------------------------ +set(SIMPLNX_TEST_ALGORITHM_PATH "0" CACHE STRING "Algorithm paths to test: 0=Both, 1=OocOnly, 2=InCoreOnly") + # ------------------------------------------------------------------------------ # Is the SimplnxCore Plugin enabled [DEFAULT=ON] # ------------------------------------------------------------------------------ @@ -255,6 +263,7 @@ if(SIMPLNX_ENABLE_MULTICORE) target_link_libraries(simplnx PUBLIC TBB::tbb) endif() + target_link_libraries(simplnx PUBLIC fmt::fmt @@ -458,6 +467,7 @@ set(SIMPLNX_HDRS ${SIMPLNX_SOURCE_DIR}/DataStructure/DynamicListArray.hpp ${SIMPLNX_SOURCE_DIR}/DataStructure/EmptyDataStore.hpp ${SIMPLNX_SOURCE_DIR}/DataStructure/EmptyListStore.hpp + ${SIMPLNX_SOURCE_DIR}/DataStructure/EmptyStringStore.hpp ${SIMPLNX_SOURCE_DIR}/DataStructure/IArray.hpp ${SIMPLNX_SOURCE_DIR}/DataStructure/IDataArray.hpp ${SIMPLNX_SOURCE_DIR}/DataStructure/IDataStore.hpp @@ -539,6 +549,7 @@ set(SIMPLNX_HDRS ${SIMPLNX_SOURCE_DIR}/Utilities/DataGroupUtilities.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/DataObjectUtilities.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/DataStoreUtilities.hpp + ${SIMPLNX_SOURCE_DIR}/Utilities/AlgorithmDispatch.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/FilePathGenerator.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/ColorTableUtilities.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/FileUtilities.hpp diff --git a/cmake/Plugin.cmake b/cmake/Plugin.cmake index 43e4691566..0f11bd11f4 100644 --- a/cmake/Plugin.cmake +++ b/cmake/Plugin.cmake @@ -383,6 +383,7 @@ function(create_simplnx_plugin_unit_test) target_compile_definitions(${UNIT_TEST_TARGET} PRIVATE SIMPLNX_BUILD_DIR="$" + SIMPLNX_TEST_ALGORITHM_PATH=${SIMPLNX_TEST_ALGORITHM_PATH} ) target_compile_options(${UNIT_TEST_TARGET} diff --git a/src/Plugins/ITKImageProcessing/src/ITKImageProcessing/Common/ITKArrayHelper.hpp b/src/Plugins/ITKImageProcessing/src/ITKImageProcessing/Common/ITKArrayHelper.hpp index 827a7f7812..d582d0cb9a 100644 --- a/src/Plugins/ITKImageProcessing/src/ITKImageProcessing/Common/ITKArrayHelper.hpp +++ b/src/Plugins/ITKImageProcessing/src/ITKImageProcessing/Common/ITKArrayHelper.hpp @@ -855,7 +855,7 @@ Result DataCheck(const DataStructure& dataStructure, const DataPa const auto& inputArray = dataStructure.getDataRefAs(inputArrayPath); const auto& inputDataStore = inputArray.getIDataStoreRef(); - if(!inputArray.getDataFormat().empty()) + if(inputArray.getStoreType() == IDataStore::StoreType::OutOfCore) { return MakeErrorResult(Constants::k_OutOfCoreDataNotSupported, fmt::format("Input Array '{}' utilizes out-of-core data. This is not supported within ITK filters.", inputArrayPath.toString())); @@ -877,7 +877,7 @@ Result> Execute(DataStr using ResultT = detail::ITKFilterFunctorResult_t; - if(!inputArray.getDataFormat().empty()) + if(inputArray.getStoreType() == IDataStore::StoreType::OutOfCore) { return MakeErrorResult(Constants::k_OutOfCoreDataNotSupported, fmt::format("Input Array '{}' utilizes out-of-core data. This is not supported within ITK filters.", inputArrayPath.toString())); } diff --git a/src/Plugins/ITKImageProcessing/test/ITKTestBase.cpp b/src/Plugins/ITKImageProcessing/test/ITKTestBase.cpp index 2773ad109a..fe394b26d1 100644 --- a/src/Plugins/ITKImageProcessing/test/ITKTestBase.cpp +++ b/src/Plugins/ITKImageProcessing/test/ITKTestBase.cpp @@ -30,7 +30,7 @@ std::string ComputeMD5HashTyped(const IDataArray& outputDataArray) usize arraySize = dataArray.getSize(); MD5 md5; - if(outputDataArray.getDataFormat().empty()) + if(outputDataArray.getIDataStoreRef().getStoreType() != IDataStore::StoreType::OutOfCore) { const T* dataPtr = dataArray.template getIDataStoreRefAs>().data(); md5.update(reinterpret_cast(dataPtr), arraySize * sizeof(T)); @@ -135,47 +135,7 @@ namespace ITKTestBase bool IsArrayInMemory(DataStructure& dataStructure, const DataPath& outputDataPath) { const auto& outputDataArray = dataStructure.getDataRefAs(outputDataPath); - DataType outputDataType = outputDataArray.getDataType(); - - switch(outputDataType) - { - case DataType::float32: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::float64: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::int8: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::uint8: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::int16: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::uint16: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::int32: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::uint32: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::int64: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::uint64: { - return dynamic_cast&>(outputDataArray).getDataFormat().empty(); - } - case DataType::boolean: { - [[fallthrough]]; - } - default: { - return {}; - } - } + return outputDataArray.getIDataStoreRef().getStoreType() != IDataStore::StoreType::OutOfCore; } //------------------------------------------------------------------------------ std::string ComputeMd5Hash(DataStructure& dataStructure, const DataPath& outputDataPath) diff --git a/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp b/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp index f3f3002789..ee2f1243d6 100644 --- a/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp @@ -44,6 +44,10 @@ TEST_CASE("OrientationAnalysis::RodriguesConvertorFilter", "[OrientationAnalysis (*exemplarData)[9] = 0.573462F; (*exemplarData)[10] = 0.655386F; (*exemplarData)[11] = 12.2066F; + (*exemplarData)[12] = 0.517892F; + (*exemplarData)[13] = 0.575435F; + (*exemplarData)[14] = 0.632979F; + (*exemplarData)[15] = 17.37815F; { // Instantiate the filter, a DataStructure object and an Arguments Object const RodriguesConvertorFilter filter; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp index dc364fca6b..84861d6480 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp @@ -1,6 +1,7 @@ #include "ComputeArrayStatistics.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/HistogramUtilities.hpp" @@ -31,7 +32,7 @@ bool CheckArraysInMemory(const nx::core::IParallelAlgorithm::AlgorithmArrays& ar continue; } - if(!arrayPtr->getIDataStoreRef().getDataFormat().empty()) + if(arrayPtr->getIDataStoreRef().getStoreType() == nx::core::IDataStore::StoreType::OutOfCore) { return false; } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp index 5cd59ae953..7c9c0eb7cd 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp @@ -302,113 +302,84 @@ const std::atomic_bool& FillBadData::getCancel() const // ============================================================================= void FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims) { - // Use negative labels for bad data regions to distinguish from positive feature IDs int64 nextLabel = -1; + const usize slabSize = static_cast(dims[0]) * static_cast(dims[1]); - const uint64 numChunks = featureIdsStore.getNumberOfChunks(); + // Two slab buffers: current Z-slab and previous Z-slab for -Z neighbor checks + std::vector curSlab(slabSize); + std::vector prevSlab(slabSize); - // Process each chunk sequentially (load, process, unload) - for(uint64 chunkIdx = 0; chunkIdx < numChunks; chunkIdx++) + for(int64 z = 0; z < dims[2]; z++) { - // Load the current chunk into memory - featureIdsStore.loadChunk(chunkIdx); + const usize slabStart = static_cast(z) * slabSize; + featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(curSlab.data(), slabSize)); - // Get chunk bounds (INCLUSIVE ranges in [Z, Y, X] order) - const auto chunkLowerBounds = featureIdsStore.getChunkLowerBounds(chunkIdx); - const auto chunkUpperBounds = featureIdsStore.getChunkUpperBounds(chunkIdx); - - // Process voxels in this chunk using scanline algorithm - // Iterate in Z-Y-X order (slowest to fastest) to maintain scanline consistency - // Note: chunk bounds are INCLUSIVE and in [Z, Y, X] order (slowest to fastest) - for(usize z = chunkLowerBounds[0]; z <= chunkUpperBounds[0]; z++) + for(int64 y = 0; y < dims[1]; y++) { - for(usize y = chunkLowerBounds[1]; y <= chunkUpperBounds[1]; y++) + for(int64 x = 0; x < dims[0]; x++) { - for(usize x = chunkLowerBounds[2]; x <= chunkUpperBounds[2]; x++) + const usize localIdx = static_cast(y) * static_cast(dims[0]) + static_cast(x); + if(curSlab[localIdx] != 0) { - // Calculate linear index for current voxel - const usize index = z * dims[0] * dims[1] + y * dims[0] + x; - - // Only process bad data voxels (FeatureId == 0) - // Skip valid feature voxels (FeatureId > 0) - if(featureIdsStore[index] != 0) - { - continue; - } + continue; + } - // Check already-processed neighbors (scanline order: -Z, -Y, -X) - // We only check "backward" neighbors because "forward" neighbors - // haven't been processed yet in the scanline order - std::vector neighborLabels; + const usize globalIdx = slabStart + localIdx; + std::vector neighborLabels; - // Check -X neighbor - if(x > 0) + // Check -X neighbor (same slab) + if(x > 0 && curSlab[localIdx - 1] == 0) + { + const usize nIdx = globalIdx - 1; + if(provisionalLabels.contains(nIdx)) { - const usize neighborIdx = index - 1; - if(provisionalLabels.contains(neighborIdx) && featureIdsStore[neighborIdx] == 0) - { - neighborLabels.push_back(provisionalLabels[neighborIdx]); - } + neighborLabels.push_back(provisionalLabels[nIdx]); } - - // Check -Y neighbor - if(y > 0) + } + // Check -Y neighbor (same slab) + if(y > 0 && curSlab[localIdx - dims[0]] == 0) + { + const usize nIdx = globalIdx - dims[0]; + if(provisionalLabels.contains(nIdx)) { - const usize neighborIdx = index - dims[0]; - if(provisionalLabels.contains(neighborIdx) && featureIdsStore[neighborIdx] == 0) - { - neighborLabels.push_back(provisionalLabels[neighborIdx]); - } + neighborLabels.push_back(provisionalLabels[nIdx]); } - - // Check -Z neighbor - if(z > 0) + } + // Check -Z neighbor (previous slab) + if(z > 0 && prevSlab[localIdx] == 0) + { + const usize nIdx = globalIdx - slabSize; + if(provisionalLabels.contains(nIdx)) { - const usize neighborIdx = index - dims[0] * dims[1]; - if(provisionalLabels.contains(neighborIdx) && featureIdsStore[neighborIdx] == 0) - { - neighborLabels.push_back(provisionalLabels[neighborIdx]); - } + neighborLabels.push_back(provisionalLabels[nIdx]); } + } - // Assign label based on neighbors - int64 assignedLabel; - if(neighborLabels.empty()) - { - // No labeled neighbors found - this is a new connected component - // Assign a new negative label and initialize in union-find - assignedLabel = nextLabel--; - unionFind.find(assignedLabel); // Initialize in union-find (creates entry) - } - else + int64 assignedLabel = 0; + if(neighborLabels.empty()) + { + assignedLabel = nextLabel--; + unionFind.find(assignedLabel); + } + else + { + assignedLabel = neighborLabels[0]; + for(usize i = 1; i < neighborLabels.size(); i++) { - // One or more labeled neighbors found - join their equivalence class - // Use the first neighbor's label as the representative - assignedLabel = neighborLabels[0]; - - // If multiple neighbors have different labels, unite them - // This handles the case where different regions merge at this voxel - for(usize i = 1; i < neighborLabels.size(); i++) + if(neighborLabels[i] != assignedLabel) { - if(neighborLabels[i] != assignedLabel) - { - unionFind.unite(assignedLabel, neighborLabels[i]); - } + unionFind.unite(assignedLabel, neighborLabels[i]); } } - - // Store the assigned label for this voxel - provisionalLabels[index] = assignedLabel; - - // Increment the size count for this label (will be accumulated to root in flatten()) - unionFind.addSize(assignedLabel, 1); } + + provisionalLabels[globalIdx] = assignedLabel; + unionFind.addSize(assignedLabel, 1); } } - } - // Flush to ensure all chunks are written back to storage - featureIdsStore.flush(); + std::swap(prevSlab, curSlab); + } } // ============================================================================= @@ -451,14 +422,9 @@ void FillBadData::phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std:: // @param maxPhase Maximum existing phase value (for new phase assignment) // ============================================================================= void FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, - const std::unordered_set& smallRegions, ChunkAwareUnionFind& unionFind, usize maxPhase) const + const std::unordered_set& /*smallRegions*/, ChunkAwareUnionFind& unionFind, usize maxPhase) const { - const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); - const SizeVec3 udims = selectedImageGeom.getDimensions(); - const uint64 numChunks = featureIdsStore.getNumberOfChunks(); - - // Collect all unique root labels and their sizes - // After flatten(), all labels point to roots and sizes are accumulated + // Classify regions by size std::unordered_map rootSizes; for(const auto& [index, label] : provisionalLabels) { @@ -469,7 +435,6 @@ void FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, } } - // Classify regions as small (need filling) or large (keep or assign to a new phase) std::unordered_set localSmallRegions; for(const auto& [root, size] : rootSizes) { @@ -479,57 +444,48 @@ void FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, } } - // Process each chunk to relabel voxels based on region classification - for(uint64 chunkIdx = 0; chunkIdx < numChunks; chunkIdx++) - { - // Load chunk into memory - featureIdsStore.loadChunk(chunkIdx); + // Process slab-by-slab, reading and writing back via bulk I/O + const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); + const SizeVec3 udims = selectedImageGeom.getDimensions(); + const usize slabSize = udims[0] * udims[1]; + std::vector slab(slabSize); - // Get chunk bounds (INCLUSIVE ranges in [Z, Y, X] order) - const auto chunkLowerBounds = featureIdsStore.getChunkLowerBounds(chunkIdx); - const auto chunkUpperBounds = featureIdsStore.getChunkUpperBounds(chunkIdx); + for(usize z = 0; z < udims[2]; z++) + { + const usize slabStart = z * slabSize; + featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(slab.data(), slabSize)); - // Iterate through all voxels in this chunk - // Note: chunk bounds are INCLUSIVE and in [Z, Y, X] order (slowest to fastest) - for(usize z = chunkLowerBounds[0]; z <= chunkUpperBounds[0]; z++) + bool slabModified = false; + for(usize localIdx = 0; localIdx < slabSize; localIdx++) { - for(usize y = chunkLowerBounds[1]; y <= chunkUpperBounds[1]; y++) + const usize globalIdx = slabStart + localIdx; + auto labelIter = provisionalLabels.find(globalIdx); + if(labelIter == provisionalLabels.end()) { - for(usize x = chunkLowerBounds[2]; x <= chunkUpperBounds[2]; x++) - { - const usize index = z * udims[0] * udims[1] + y * udims[0] + x; - - // Check if this voxel was labeled as bad data in Phase 1 - auto labelIter = provisionalLabels.find(index); - if(labelIter != provisionalLabels.end()) - { - // Find the root label for this voxel's connected component - int64 root = unionFind.find(labelIter->second); + continue; + } - if(localSmallRegions.contains(root)) - { - // Small region - mark with -1 for filling in Phase 4 - featureIdsStore[index] = -1; - } - else - { - // Large region - keep as bad data (0) or assign to a new phase - featureIdsStore[index] = 0; - - // Optionally assign large bad data regions to a new phase - if(m_InputValues->storeAsNewPhase && cellPhasesPtr != nullptr) - { - (*cellPhasesPtr)[index] = static_cast(maxPhase) + 1; - } - } - } + int64 root = unionFind.find(labelIter->second); + if(localSmallRegions.contains(root)) + { + slab[localIdx] = -1; + } + else + { + slab[localIdx] = 0; + if(m_InputValues->storeAsNewPhase && cellPhasesPtr != nullptr) + { + (*cellPhasesPtr)[globalIdx] = static_cast(maxPhase) + 1; } } + slabModified = true; } - } - // Write all chunks back to storage - featureIdsStore.flush(); + if(slabModified) + { + featureIdsStore.copyFromBuffer(slabStart, nonstd::span(slab.data(), slabSize)); + } + } } // ============================================================================= @@ -695,12 +651,10 @@ void FillBadData::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore // ============================================================================= Result<> FillBadData::operator()() const { - // Get feature IDs array and image geometry auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath)->getDataStoreRef(); const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); const SizeVec3 udims = selectedImageGeom.getDimensions(); - // Convert dimensions to signed integers for offset calculations std::array dims = { static_cast(udims[0]), static_cast(udims[1]), @@ -709,15 +663,11 @@ Result<> FillBadData::operator()() const const usize totalPoints = featureIdsStore.getNumberOfTuples(); - // Get cell phases array if we need to assign large regions to a new phase Int32Array* cellPhasesPtr = nullptr; usize maxPhase = 0; - if(m_InputValues->storeAsNewPhase) { cellPhasesPtr = m_DataStructure.getDataAs(m_InputValues->cellPhasesArrayPath); - - // Find the maximum existing phase value for(usize i = 0; i < totalPoints; i++) { if((*cellPhasesPtr)[i] > maxPhase) @@ -727,35 +677,37 @@ Result<> FillBadData::operator()() const } } - // Count the number of existing features for array sizing usize numFeatures = 0; - for(usize i = 0; i < totalPoints; i++) { - int32 featureName = featureIdsStore[i]; - if(featureName > numFeatures) + const usize bufSize = 65536; + std::vector buf(bufSize); + for(usize offset = 0; offset < totalPoints; offset += bufSize) { - numFeatures = featureName; + const usize count = std::min(bufSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); + for(usize i = 0; i < count; i++) + { + if(buf[i] > static_cast(numFeatures)) + { + numFeatures = buf[i]; + } + } } } - // Initialize data structures for chunk-aware connected component labeling - ChunkAwareUnionFind unionFind; // Tracks label equivalences and sizes - std::unordered_map provisionalLabels; // Maps voxel index to provisional label - std::unordered_set smallRegions; // Set of small region roots (unused currently) + ChunkAwareUnionFind unionFind; + std::unordered_map provisionalLabels; + std::unordered_set smallRegions; - // Phase 1: Chunk-Sequential Connected Component Labeling m_MessageHandler({IFilter::Message::Type::Info, "Phase 1/4: Labeling connected components..."}); phaseOneCCL(featureIdsStore, unionFind, provisionalLabels, dims); - // Phase 2: Global Resolution of equivalences m_MessageHandler({IFilter::Message::Type::Info, "Phase 2/4: Resolving region equivalences..."}); phaseTwoGlobalResolution(unionFind, smallRegions); - // Phase 3: Relabeling based on region size classification m_MessageHandler({IFilter::Message::Type::Info, "Phase 3/4: Classifying region sizes..."}); phaseThreeRelabeling(featureIdsStore, cellPhasesPtr, provisionalLabels, smallRegions, unionFind, maxPhase); - // Phase 4: Iterative morphological fill m_MessageHandler({IFilter::Message::Type::Info, "Phase 4/4: Filling small defects..."}); phaseFourIterativeFill(featureIdsStore, dims, numFeatures); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp index e0016d3d54..10ffc4c42f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp @@ -1608,11 +1608,6 @@ void QuickSurfaceMesh::generateTripleLines() // Cycle through again assigning coordinates to each node and assigning node numbers and feature labels to each triangle ParallelData3DAlgorithm algorithm; algorithm.setRange(Range3D(xP - 1, yP - 1, zP - 1)); - if(featureIds.getChunkShape().has_value()) - { - const auto chunkShape = featureIds.getChunkShape().value(); - algorithm.setChunkSize(Range3D(chunkShape[0], chunkShape[1], chunkShape[2])); - } algorithm.setParallelizationEnabled(false); algorithm.execute(GenerateTripleLinesImpl(imageGeom, featureIds, vertexMap, edgeMap)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp index 6270f4c036..5cae085711 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp @@ -141,7 +141,7 @@ struct WriteVtkDataArrayFunctor void operator()(FILE* outputFile, bool binary, DataStructure& dataStructure, const DataPath& arrayPath, const IFilter::MessageHandler& messageHandler) { auto* dataArray = dataStructure.getDataAs>(arrayPath); - auto& dataStore = dataArray->template getIDataStoreRefAs>(); + auto& dataStore = dataArray->getDataStoreRef(); messageHandler(IFilter::Message::Type::Info, fmt::format("Writing Cell Data {}", arrayPath.getTargetName())); @@ -161,16 +161,37 @@ struct WriteVtkDataArrayFunctor fprintf(outputFile, "LOOKUP_TABLE default\n"); if(binary) { - if constexpr(endian::little == endian::native) + // Read data into buffer, byte-swap in memory, and write. + // This avoids modifying the DataStore (critical for OOC stores + // where per-element byte-swap via setValue would be very slow). + constexpr usize k_ChunkSize = 4096; + for(usize offset = 0; offset < totalElements; offset += k_ChunkSize) { - dataArray->byteSwapElements(); + usize count = std::min(k_ChunkSize, totalElements - offset); + if constexpr(std::is_same_v) + { + std::vector buf(count); + for(usize i = 0; i < count; i++) + { + buf[i] = dataStore.getValue(offset + i) ? 1 : 0; + } + fwrite(buf.data(), sizeof(uint8), count, outputFile); + } + else + { + std::vector buf(count); + dataStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); + if constexpr(endian::little == endian::native) + { + for(usize i = 0; i < count; i++) + { + buf[i] = nx::core::byteswap(buf[i]); + } + } + fwrite(buf.data(), sizeof(T), count, outputFile); + } } - fwrite(dataStore.data(), sizeof(T), totalElements, outputFile); fprintf(outputFile, "\n"); - if constexpr(endian::little == endian::native) - { - dataArray->byteSwapElements(); - } } else { @@ -184,7 +205,7 @@ struct WriteVtkDataArrayFunctor } if(useIntCast) { - buffer.append(fmt::format(" {:d}", static_cast(dataStore[i]))); + buffer.append(fmt::format(" {:d}", static_cast(dataStore.getValue(i)))); } else if constexpr(std::is_floating_point_v) { diff --git a/src/simplnx/Core/Preferences.cpp b/src/simplnx/Core/Preferences.cpp index aa12ef5776..404ea32451 100644 --- a/src/simplnx/Core/Preferences.cpp +++ b/src/simplnx/Core/Preferences.cpp @@ -22,10 +22,10 @@ namespace nx::core namespace { constexpr int64 k_LargeDataSize = 1073741824; // 1 GB -constexpr StringLiteral k_LargeDataFormat = ""; constexpr StringLiteral k_Plugin_Key = "plugins"; constexpr StringLiteral k_DefaultFileName = "preferences.json"; constexpr int64 k_ReducedDataStructureSize = 3221225472; // 3 GB +constexpr uint32 k_OocRangeScanTimeoutSecondsDefault = 30; constexpr int32 k_FailedToCreateDirectory_Code = -585; constexpr int32 k_FileDoesNotExist_Code = -586; @@ -77,7 +77,6 @@ void Preferences::setDefaultValues() m_DefaultValues[k_Plugin_Key] = nlohmann::json::object(); m_DefaultValues[k_LargeDataSize_Key] = k_LargeDataSize; - m_DefaultValues[k_PreferredLargeDataFormat_Key] = k_LargeDataFormat; { // Set a default value for out-of-core temp directory. @@ -92,6 +91,8 @@ void Preferences::setDefaultValues() #else m_DefaultValues[k_ForceOocData_Key] = false; #endif + + m_DefaultValues[k_OocRangeScanTimeoutSeconds_Key] = k_OocRangeScanTimeoutSecondsDefault; } std::string Preferences::defaultLargeDataFormat() const @@ -107,11 +108,26 @@ void Preferences::setDefaultLargeDataFormat(std::string dataFormat) std::string Preferences::largeDataFormat() const { - return valueAs(k_PreferredLargeDataFormat_Key); + auto formatJson = value(k_PreferredLargeDataFormat_Key); + if(formatJson.is_null() || !formatJson.is_string()) + { + return {}; + } + return formatJson.get(); } void Preferences::setLargeDataFormat(std::string dataFormat) { - m_Values[k_PreferredLargeDataFormat_Key] = dataFormat; + if(dataFormat.empty()) + { + // Remove the key so the default (set by plugins) can take effect. + // An empty string means "not configured", not "in-core" — use + // k_InMemoryFormat for an explicit in-core choice. + m_Values.erase(k_PreferredLargeDataFormat_Key); + } + else + { + m_Values[k_PreferredLargeDataFormat_Key] = dataFormat; + } checkUseOoc(); } @@ -255,6 +271,18 @@ Result<> Preferences::loadFromFile(const std::filesystem::path& filepath) m_Values = parsedResult; + // Migrate legacy format strings from saved preferences: + // - Empty string: legacy "not configured" state → remove so plugin defaults take effect + // - "In-Memory": legacy explicit in-core choice → remove so plugin defaults take effect + if(m_Values.contains(k_PreferredLargeDataFormat_Key) && m_Values[k_PreferredLargeDataFormat_Key].is_string()) + { + const std::string savedFormat = m_Values[k_PreferredLargeDataFormat_Key].get(); + if(savedFormat.empty() || savedFormat == "In-Memory") + { + m_Values.erase(k_PreferredLargeDataFormat_Key); + } + } + checkUseOoc(); updateMemoryDefaults(); return {}; @@ -262,7 +290,14 @@ Result<> Preferences::loadFromFile(const std::filesystem::path& filepath) void Preferences::checkUseOoc() { - m_UseOoc = !value(k_PreferredLargeDataFormat_Key).get().empty(); + auto formatJson = value(k_PreferredLargeDataFormat_Key); + if(formatJson.is_null() || !formatJson.is_string()) + { + m_UseOoc = false; + return; + } + const std::string format = formatJson.get(); + m_UseOoc = !format.empty() && format != k_InMemoryFormat; } bool Preferences::useOocData() const @@ -320,4 +355,27 @@ void Preferences::setOocTempDirectory(const std::string& path) plugin->setOocTempDirectory(path); } } + +uint32 Preferences::oocRangeScanTimeoutSeconds() const +{ + return value(k_OocRangeScanTimeoutSeconds_Key).get(); +} + +void Preferences::setOocRangeScanTimeoutSeconds(uint32 seconds) +{ + setValue(k_OocRangeScanTimeoutSeconds_Key, seconds); +} + +uint64 Preferences::oocMemoryBudgetBytes() const +{ + // Default: 8 GB (the application will set the real default from + // OocMemoryBudgetManager::defaultBudgetBytes() on startup) + static constexpr uint64 k_DefaultBudget = 8ULL * 1024 * 1024 * 1024; + return m_Values.value(k_OocMemoryBudgetBytes_Key, k_DefaultBudget); +} + +void Preferences::setOocMemoryBudgetBytes(uint64 bytes) +{ + m_Values[k_OocMemoryBudgetBytes_Key] = bytes; +} } // namespace nx::core diff --git a/src/simplnx/Core/Preferences.hpp b/src/simplnx/Core/Preferences.hpp index cec687aef0..5c02226266 100644 --- a/src/simplnx/Core/Preferences.hpp +++ b/src/simplnx/Core/Preferences.hpp @@ -24,11 +24,14 @@ class SIMPLNX_EXPORT Preferences friend class AbstractPlugin; public: - static inline constexpr StringLiteral k_LargeDataSize_Key = "large_data_size"; // bytes - static inline constexpr StringLiteral k_PreferredLargeDataFormat_Key = "large_data_format"; // string - static inline constexpr StringLiteral k_LargeDataStructureSize_Key = "large_datastructure_size"; // bytes - static inline constexpr StringLiteral k_ForceOocData_Key = "force_ooc_data"; // boolean - static inline constexpr nx::core::StringLiteral k_OoCTempDirectory_ID = "ooc_temp_directory"; // Out-of-Core temp directory + static inline constexpr StringLiteral k_LargeDataSize_Key = "large_data_size"; // bytes + static inline constexpr StringLiteral k_PreferredLargeDataFormat_Key = "large_data_format"; // string + static inline constexpr StringLiteral k_InMemoryFormat = "Simplnx-Default-In-Memory"; // explicit in-core; empty string means "not configured" + static inline constexpr StringLiteral k_LargeDataStructureSize_Key = "large_datastructure_size"; // bytes + static inline constexpr StringLiteral k_ForceOocData_Key = "force_ooc_data"; // boolean + static inline constexpr nx::core::StringLiteral k_OoCTempDirectory_ID = "ooc_temp_directory"; // Out-of-Core temp directory + static inline constexpr StringLiteral k_OocRangeScanTimeoutSeconds_Key = "ooc_range_scan_timeout_seconds"; // uint32, seconds + static inline constexpr StringLiteral k_OocMemoryBudgetBytes_Key = "ooc_memory_budget_bytes"; // uint64, bytes /** * @brief Returns the default file path for storing preferences based on the application name. @@ -249,6 +252,36 @@ class SIMPLNX_EXPORT Preferences */ void setOocTempDirectory(const std::string& path); + /** + * @brief Gets the timeout (in seconds) for the background OOC range scan. + * @return Timeout in seconds (default 30) + */ + uint32 oocRangeScanTimeoutSeconds() const; + + /** + * @brief Sets the timeout (in seconds) for the background OOC range scan. + * @param seconds Timeout value in seconds + */ + void setOocRangeScanTimeoutSeconds(uint32 seconds); + + /** + * @brief Gets the total memory budget for all OOC caching (chunk cache, stride cache, partition cache). + * + * The budget manager distributes this across subsystems via global LRU eviction. + * The default (8 GB) is a safe fallback; the SimplnxOoc plugin overrides this on + * startup with OocMemoryBudgetManager::defaultBudgetBytes() (50% of system RAM) + * if the user has not saved an explicit preference. + * + * @return Budget in bytes (default 8 GB) + */ + uint64 oocMemoryBudgetBytes() const; + + /** + * @brief Sets the total memory budget for all OOC caching. + * @param bytes Budget in bytes + */ + void setOocMemoryBudgetBytes(uint64 bytes); + protected: /** * @brief Initializes all default preference values for the application. diff --git a/src/simplnx/DataStructure/AbstractDataStore.hpp b/src/simplnx/DataStructure/AbstractDataStore.hpp index 2996fdce12..43b7d1e574 100644 --- a/src/simplnx/DataStructure/AbstractDataStore.hpp +++ b/src/simplnx/DataStructure/AbstractDataStore.hpp @@ -410,6 +410,27 @@ class AbstractDataStore : public IDataStore */ virtual void setValue(usize index, value_type value) = 0; + /** + * @brief Copies a contiguous range of values from this data store into the + * provided buffer. The buffer must be large enough to hold the requested + * range. Each store subclass implements its own optimal version: in-memory + * stores use std::copy; out-of-core stores use bulk chunk I/O. + * + * @param startIndex The starting flat element index to read from + * @param buffer A span to receive the copied values (buffer.size() determines count) + */ + virtual void copyIntoBuffer(usize startIndex, nonstd::span buffer) const = 0; + + /** + * @brief Copies values from the provided buffer into a contiguous range of + * this data store. Each store subclass implements its own optimal version: + * in-memory stores use std::copy; out-of-core stores use bulk chunk I/O. + * + * @param startIndex The starting flat element index to write to + * @param buffer A span containing the values to copy into the store + */ + virtual void copyFromBuffer(usize startIndex, nonstd::span buffer) = 0; + /** * @brief Returns the value found at the specified index of the DataStore. * This cannot be used to edit the value found at the specified index. @@ -804,132 +825,6 @@ class AbstractDataStore : public IDataStore return getValue(index); } - std::optional getChunkShape() const override - { - return {}; - } - - /** - * @brief Returns the data for a particular data chunk. Returns an empty span if the data is not chunked. - * @param chunkPosition - * @return chunk data as span - */ - virtual std::vector getChunkValues(const ShapeType& chunkPosition) const - { - return {}; - } - - /** - * @brief Returns the number of chunks used to store the data. - * @return uint64 - */ - virtual uint64 getNumberOfChunks() const = 0; - // { - // return 1; - // } - - /** - * @brief Returns the number of elements in the specified chunk index. - * @param flatChunkIndex - * @return - */ - virtual uint64 getChunkSize(uint64 flatChunkIndex) const - { - if(flatChunkIndex >= getNumberOfChunks()) - { - return 0; - } - return size(); - } - - /** - * @brief Returns the Smallest N-Dimensional tuple position included in the - * specified chunk. - * @param flatChunkIndex - * @return ShapeType - */ - virtual ShapeType getChunkLowerBounds(uint64 flatChunkIndex) const = 0; - - /** - * @brief Returns the largest N-Dimensional tuple position included in the - * specified chunk. - * @param flatChunkIndex - * @return ShapeType - */ - virtual ShapeType getChunkUpperBounds(uint64 flatChunkIndex) const = 0; - - /** - * @brief Returns the tuple shape for the specified chunk. - * Returns an empty vector if the chunk is out of bounds. - * @param flatChunkIndex - * @return std::vector chunk tuple shape - */ - virtual ShapeType getChunkTupleShape(uint64 flatChunkIndex) const - { - if(flatChunkIndex >= getNumberOfChunks()) - { - return ShapeType(); - } - auto lowerBounds = getChunkLowerBounds(flatChunkIndex); - auto upperBounds = getChunkUpperBounds(flatChunkIndex); - - const usize tupleCount = lowerBounds.size(); - ShapeType chunkTupleShape(tupleCount); - for(usize i = 0; i < tupleCount; i++) - { - chunkTupleShape[i] = upperBounds[i] - lowerBounds[i] + 1; - } - return chunkTupleShape; - } - - /** - * @brief Returns a vector containing the tuple extents for a specified chunk. - * The returned values are formatted as [min, max] in the order of the tuple - * dimensions. For instance, a single chunk with tuple dimensions {X, Y, Z} - * will result in an extent of [0, X-1, 0, Y-1, 0, Z-1]. - * Returns an empty vector if the chunk requested is beyond the scope of the - * available chunks. - * @param flatChunkIndex - * @return std::vector extents - */ - std::vector getChunkExtents(uint64 flatChunkIndex) const - { - if(flatChunkIndex >= getNumberOfChunks()) - { - return std::vector(); - } - - usize tupleDims = getTupleShape().size(); - std::vector extents(tupleDims * 2); - - auto upperBounds = getChunkUpperBounds(flatChunkIndex); - auto lowerBounds = getChunkLowerBounds(flatChunkIndex); - - for(usize i = 0; i < tupleDims; i++) - { - extents[i * 2] = lowerBounds[i]; - extents[i * 2 + 1] = upperBounds[i]; - } - - return extents; - } - - /** - * @brief Makes sure the target chunk is loaded in memory. - * This method does nothing for in-memory DataStores. - * @param flatChunkIndex - */ - virtual void loadChunk(uint64 flatChunkIndex) - { - } - - /** - * @brief Creates and returns an in-memory AbstractDataStore from a copy of the data - * from the specified chunk. - * @param flatChunkIndex - */ - virtual std::unique_ptr> convertChunkToDataStore(uint64 flatChunkIndex) const = 0; - /** * @brief Flushes the data store to its respective target. * In-memory DataStores are not affected. diff --git a/src/simplnx/DataStructure/AbstractStringStore.hpp b/src/simplnx/DataStructure/AbstractStringStore.hpp index eb2b3760c8..1af0cc2e48 100644 --- a/src/simplnx/DataStructure/AbstractStringStore.hpp +++ b/src/simplnx/DataStructure/AbstractStringStore.hpp @@ -2,6 +2,7 @@ #include "simplnx/Common/Aliases.hpp" #include "simplnx/Common/Types.hpp" +#include "simplnx/simplnx_export.hpp" #include #include @@ -9,7 +10,23 @@ namespace nx::core { -class AbstractStringStore +/** + * @class AbstractStringStore + * @brief Abstract base class for string storage backends used by StringArray. + * + * AbstractStringStore defines the interface for storing and accessing an + * ordered collection of strings, organized by tuple shape. Concrete + * subclasses include: + * + * - **StringStore** -- The real, in-memory store that owns a + * `std::vector` and supports full read/write access. + * - **EmptyStringStore** -- A metadata-only placeholder that records + * tuple shape but holds no data. All data access methods throw. + * + * The isPlaceholder() virtual method allows callers to distinguish + * between these two cases without dynamic_cast. + */ +class SIMPLNX_EXPORT AbstractStringStore { public: using value_type = std::string; @@ -323,7 +340,12 @@ class AbstractStringStore using iterator = Iterator; using const_iterator = ConstIterator; - ~AbstractStringStore() = default; + /** + * @brief Virtual destructor. Ensures correct cleanup when deleting through + * a base class pointer, which is the normal ownership pattern since + * StringArray holds an AbstractStringStore via std::unique_ptr. + */ + virtual ~AbstractStringStore() = default; /** * @brief Creates a deep copy of this AbstractStringStore. @@ -343,6 +365,23 @@ class AbstractStringStore */ virtual bool empty() const = 0; + /** + * @brief Checks whether this store is a metadata-only placeholder that + * holds no real string data. + * + * This method exists so that import/backfill logic can identify which + * StringArray objects in a DataStructure still need their data loaded + * without resorting to dynamic_cast. The two concrete subclasses return + * fixed values: + * + * - **StringStore::isPlaceholder()** returns `false` (data is present). + * - **EmptyStringStore::isPlaceholder()** returns `true` (no data; + * accessing elements will throw). + * + * @return true if this store is a placeholder with no accessible data + */ + virtual bool isPlaceholder() const = 0; + /** * @brief Returns the number of tuples in the StringStore. * @return usize diff --git a/src/simplnx/DataStructure/DataStore.hpp b/src/simplnx/DataStructure/DataStore.hpp index cb0e5cb5c2..cb75fa26a4 100644 --- a/src/simplnx/DataStructure/DataStore.hpp +++ b/src/simplnx/DataStructure/DataStore.hpp @@ -58,8 +58,8 @@ class DataStore : public AbstractDataStore : parent_type() , m_ComponentShape(componentShape) , m_TupleShape(tupleShape) - , m_NumComponents(std::accumulate(m_ComponentShape.cbegin(), m_ComponentShape.cend(), static_cast(1), std::multiplies<>())) - , m_NumTuples(std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumComponents(std::accumulate(m_ComponentShape.cbegin(), m_ComponentShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumTuples(std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>())) , m_InitValue(initValue) { resizeTuples(m_TupleShape); @@ -80,8 +80,8 @@ class DataStore : public AbstractDataStore , m_ComponentShape(std::move(componentShape)) , m_TupleShape(std::move(tupleShape)) , m_Data(std::move(buffer)) - , m_NumComponents(std::accumulate(m_ComponentShape.cbegin(), m_ComponentShape.cend(), static_cast(1), std::multiplies<>())) - , m_NumTuples(std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumComponents(std::accumulate(m_ComponentShape.cbegin(), m_ComponentShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumTuples(std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>())) { // Because no init value is passed into the constructor, we will use a "mudflap" style value that is easy to debug. m_InitValue = GetMudflap(); @@ -240,7 +240,7 @@ class DataStore : public AbstractDataStore auto oldSize = this->getSize(); // Calculate the total number of values in the new array m_TupleShape = tupleShape; - m_NumTuples = std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>()); + m_NumTuples = std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>()); usize newSize = getNumberOfComponents() * m_NumTuples; @@ -299,6 +299,26 @@ class DataStore : public AbstractDataStore m_Data.get()[index] = value; } + void copyIntoBuffer(usize startIndex, nonstd::span buffer) const override + { + const usize count = buffer.size(); + if(startIndex + count > this->getSize()) + { + throw std::out_of_range(fmt::format("DataStore::copyIntoBuffer: range [{}, {}) exceeds size {}", startIndex, startIndex + count, this->getSize())); + } + std::copy(m_Data.get() + startIndex, m_Data.get() + startIndex + count, buffer.data()); + } + + void copyFromBuffer(usize startIndex, nonstd::span buffer) override + { + const usize count = buffer.size(); + if(startIndex + count > this->getSize()) + { + throw std::out_of_range(fmt::format("DataStore::copyFromBuffer: range [{}, {}) exceeds size {}", startIndex, startIndex + count, this->getSize())); + } + std::copy(buffer.begin(), buffer.end(), m_Data.get() + startIndex); + } + /** * @brief Returns the value found at the specified index of sthe DataStore. * This cannot be used to edit the value found at the specified index. @@ -579,80 +599,12 @@ class DataStore : public AbstractDataStore return dataset.writeSpan(dims, span); } - /** - * @brief Creates and returns an in-memory AbstractDataStore from a copy of the data - * from the specified chunk. - * @param flatChunkIndex - */ - std::unique_ptr> convertChunkToDataStore(uint64 flatChunkIndex) const override - { - if(flatChunkIndex >= this->getNumberOfChunks()) - { - return nullptr; - } - - std::unique_ptr dataWrapper = std::make_unique_for_overwrite(this->getSize()); - std::copy(this->begin(), this->end(), dataWrapper.get()); - - return std::make_unique>(std::move(dataWrapper), this->getTupleShape(), this->getComponentShape()); - } - - /** - * @brief Returns the number of chunks used to store the data. - * For in-memory DataStore, this is always 1. - * @return uint64 The number of chunks (always 1 for in-memory storage) - */ - uint64 getNumberOfChunks() const override - { - return 1; - } - - /** - * @brief Returns the Smallest N-Dimensional tuple position included in the - * specified chunk. - * @param flatChunkIndex - * @return ShapeType - */ - ShapeType getChunkLowerBounds(uint64 flatChunkIndex) const override - { - if(flatChunkIndex >= getNumberOfChunks()) - { - return ShapeType(); - } - usize tupleDims = getTupleShape().size(); - - ShapeType lowerBounds(tupleDims); - std::fill(lowerBounds.begin(), lowerBounds.end(), 0); - return lowerBounds; - } - - /** - * @brief Returns the largest N-Dimensional tuple position included in the - * specified chunk. - * @param flatChunkIndex - * @return ShapeType - */ - ShapeType getChunkUpperBounds(uint64 flatChunkIndex) const override - { - if(flatChunkIndex >= getNumberOfChunks()) - { - return ShapeType(); - } - - ShapeType upperBounds(getTupleShape()); - for(auto& value : upperBounds) - { - value -= 1; - } - return upperBounds; - } - private: ShapeType m_ComponentShape; ShapeType m_TupleShape; std::unique_ptr m_Data = nullptr; - size_t m_NumComponents = {0}; - size_t m_NumTuples = {0}; + usize m_NumComponents = {0}; + usize m_NumTuples = {0}; std::optional m_InitValue; }; diff --git a/src/simplnx/DataStructure/EmptyDataStore.hpp b/src/simplnx/DataStructure/EmptyDataStore.hpp index dd2cb44a16..cbe280719e 100644 --- a/src/simplnx/DataStructure/EmptyDataStore.hpp +++ b/src/simplnx/DataStructure/EmptyDataStore.hpp @@ -38,8 +38,8 @@ class EmptyDataStore : public AbstractDataStore EmptyDataStore(const ShapeType& tupleShape, const ShapeType& componentShape, std::string dataFormat = "") : m_ComponentShape(componentShape) , m_TupleShape(tupleShape) - , m_NumComponents(std::accumulate(m_ComponentShape.cbegin(), m_ComponentShape.cend(), static_cast(1), std::multiplies<>())) - , m_NumTuples(std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumComponents(std::accumulate(m_ComponentShape.cbegin(), m_ComponentShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumTuples(std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>())) , m_DataFormat(dataFormat) { } @@ -85,7 +85,7 @@ class EmptyDataStore : public AbstractDataStore * @brief Returns the target tuple getSize. * @return usize */ - size_t getNumberOfComponents() const override + usize getNumberOfComponents() const override { return m_NumComponents; } @@ -114,7 +114,7 @@ class EmptyDataStore : public AbstractDataStore */ IDataStore::StoreType getStoreType() const override { - return m_DataFormat.empty() ? IDataStore::StoreType::Empty : IDataStore::StoreType::EmptyOutOfCore; + return IDataStore::StoreType::Empty; } /** @@ -158,6 +158,16 @@ class EmptyDataStore : public AbstractDataStore throw std::runtime_error("EmptyDataStore::setValue() is not implemented"); } + void copyIntoBuffer(usize startIndex, nonstd::span buffer) const override + { + throw std::runtime_error("EmptyDataStore::copyIntoBuffer() is not implemented"); + } + + void copyFromBuffer(usize startIndex, nonstd::span buffer) override + { + throw std::runtime_error("EmptyDataStore::copyFromBuffer() is not implemented"); + } + /** * @brief Throws an exception because this should never be called. The * EmptyDataStore class contains no data other than its target getSize. @@ -347,50 +357,11 @@ class EmptyDataStore : public AbstractDataStore return MakeErrorResult(-42350, "Cannot write data from an EmptyDataStore"); } - /** - * @brief Creates and returns an in-memory AbstractDataStore from a copy of the data - * from the specified chunk. - * @param flatChunkIndex - */ - std::unique_ptr> convertChunkToDataStore(uint64 flatChunkIndex) const override - { - return nullptr; - } - - /** - * @brief Returns empty bounds because EmptyDataStore has no chunks. - * @param flatChunkIndex The chunk index (unused) - * @return ShapeType Empty shape vector - */ - ShapeType getChunkLowerBounds(uint64 flatChunkIndex) const override - { - return {}; - } - - /** - * @brief Returns empty bounds because EmptyDataStore has no chunks. - * @param flatChunkIndex The chunk index (unused) - * @return ShapeType Empty shape vector - */ - ShapeType getChunkUpperBounds(uint64 flatChunkIndex) const override - { - return {}; - } - - /** - * @brief Returns the number of chunks in the EmptyDataStore. - * @return uint64 Always returns 0 because EmptyDataStore has no data - */ - uint64 getNumberOfChunks() const override - { - return 0; - } - private: ShapeType m_ComponentShape; ShapeType m_TupleShape; - size_t m_NumComponents = {0}; - size_t m_NumTuples = {0}; + usize m_NumComponents = {0}; + usize m_NumTuples = {0}; std::string m_DataFormat = ""; }; } // namespace nx::core diff --git a/src/simplnx/DataStructure/EmptyStringStore.hpp b/src/simplnx/DataStructure/EmptyStringStore.hpp new file mode 100644 index 0000000000..14ffcbc4df --- /dev/null +++ b/src/simplnx/DataStructure/EmptyStringStore.hpp @@ -0,0 +1,207 @@ +#pragma once + +#include "AbstractStringStore.hpp" + +#include +#include +#include + +namespace nx::core +{ +/** + * @class EmptyStringStore + * @brief A metadata-only placeholder for AbstractStringStore, analogous to + * EmptyDataStore for numeric arrays. + * + * EmptyStringStore records tuple shape (number and layout of strings) but + * holds no actual string data. It exists because: + * + * 1. **Preflight-style imports:** When loading a .dream3d file's + * DataStructure in metadata-only mode (e.g., for file inspection or + * pipeline validation), StringArray objects need a store that reports + * correct tuple counts without allocating or reading string data. + * + * 2. **Out-of-core workflows:** When the OOC import path builds the + * DataStructure skeleton, string arrays are initially populated with + * EmptyStringStore instances. A subsequent backfill step replaces each + * EmptyStringStore with a real StringStore that contains the loaded + * data. + * + * All data access methods (operator[], at, getValue, setValue, operator=) + * throw std::runtime_error to fail fast if code accidentally tries to read + * or write string data before the backfill step has run. + * + * @see StringStore The concrete store that holds real string data. + * @see EmptyDataStore The equivalent placeholder for numeric DataArrays. + */ +class SIMPLNX_EXPORT EmptyStringStore : public AbstractStringStore +{ +public: + /** + * @brief Default constructor. + */ + EmptyStringStore() = default; + + /** + * @brief Constructs an EmptyStringStore with the specified tuple shape. + * @param tupleShape The shape of the tuple dimensions + */ + EmptyStringStore(const ShapeType& tupleShape) + : AbstractStringStore() + , m_TupleShape(tupleShape) + , m_NumTuples(std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>())) + { + } + + /** + * @brief Copy constructor. + * @param rhs The EmptyStringStore to copy from + */ + EmptyStringStore(const EmptyStringStore& rhs) = default; + + /** + * @brief Move constructor. + * @param rhs The EmptyStringStore to move from + */ + EmptyStringStore(EmptyStringStore&& rhs) = default; + + ~EmptyStringStore() override = default; + + /** + * @brief Creates a deep copy of this EmptyStringStore. + * @return std::unique_ptr Unique pointer to the deep copy + */ + std::unique_ptr deepCopy() const override + { + return std::make_unique(*this); + } + + /** + * @brief Returns the total number of strings in the store (equal to the number of tuples). + * @return usize The number of strings + */ + usize size() const override + { + return m_NumTuples; + } + + /** + * @brief Returns whether the string store is empty. + * @return bool True if the store has no strings, false otherwise + */ + bool empty() const override + { + return m_NumTuples == 0; + } + + /** + * @brief Returns the number of tuples in the EmptyStringStore. + * @return usize + */ + usize getNumberOfTuples() const override + { + return m_NumTuples; + } + + /** + * @brief Returns the dimensions of the Tuples + * @return + */ + const ShapeType& getTupleShape() const override + { + return m_TupleShape; + } + + /** + * @brief Resizes the string store to the specified tuple shape. + * @param tupleShape The new shape of the tuple dimensions + */ + void resizeTuples(const ShapeType& tupleShape) override + { + m_TupleShape = tupleShape; + m_NumTuples = std::accumulate(m_TupleShape.cbegin(), m_TupleShape.cend(), static_cast(1), std::multiplies<>()); + } + + /** + * @brief Returns true because EmptyStringStore is a metadata-only + * placeholder that holds no actual string data. + * + * Code that needs to distinguish between a real StringStore (which has + * accessible data) and an EmptyStringStore (which will throw on access) + * should call isPlaceholder() rather than using dynamic_cast. This is + * used by the backfill/import logic to identify which stores still need + * their data loaded. + * + * @return true Always returns true for EmptyStringStore. + */ + bool isPlaceholder() const override + { + return true; + } + + /** + * @brief Throws an error because EmptyStringStore has no data. + * @param index The index (unused) + * @throw std::runtime_error Always throws because EmptyStringStore has no data + */ + reference operator[](usize index) override + { + throw std::runtime_error("EmptyStringStore::operator[] called on placeholder store - data not loaded yet"); + } + + /** + * @brief Throws an error because EmptyStringStore has no data. + * @param index The index (unused) + * @throw std::runtime_error Always throws because EmptyStringStore has no data + */ + const_reference operator[](usize index) const override + { + throw std::runtime_error("EmptyStringStore::operator[] called on placeholder store - data not loaded yet"); + } + + /** + * @brief Throws an error because EmptyStringStore has no data. + * @param index The index (unused) + * @throw std::runtime_error Always throws because EmptyStringStore has no data + */ + const_reference at(usize index) const override + { + throw std::runtime_error("EmptyStringStore::at() called on placeholder store - data not loaded yet"); + } + + /** + * @brief Throws an error because EmptyStringStore has no data. + * @param index The index (unused) + * @throw std::runtime_error Always throws because EmptyStringStore has no data + */ + const_reference getValue(usize index) const override + { + throw std::runtime_error("EmptyStringStore::getValue() called on placeholder store - data not loaded yet"); + } + + /** + * @brief Throws an error because EmptyStringStore has no data. + * @param index The index (unused) + * @param value The value to set (unused) + * @throw std::runtime_error Always throws because EmptyStringStore has no data + */ + void setValue(usize index, const value_type& value) override + { + throw std::runtime_error("EmptyStringStore::setValue() called on placeholder store - data not loaded yet"); + } + + /** + * @brief Throws an error because EmptyStringStore cannot accept data assignments. + * @param values Vector of strings to assign (unused) + * @throw std::runtime_error Always throws because EmptyStringStore has no data + */ + AbstractStringStore& operator=(const std::vector& values) override + { + throw std::runtime_error("EmptyStringStore::operator= called on placeholder store - data not loaded yet"); + } + +private: + ShapeType m_TupleShape; + usize m_NumTuples = 0; +}; +} // namespace nx::core diff --git a/src/simplnx/DataStructure/IDataStore.hpp b/src/simplnx/DataStructure/IDataStore.hpp index 4f644f0227..1ac027835d 100644 --- a/src/simplnx/DataStructure/IDataStore.hpp +++ b/src/simplnx/DataStructure/IDataStore.hpp @@ -6,8 +6,8 @@ #include #include +#include #include -#include #include #include @@ -27,8 +27,7 @@ class SIMPLNX_EXPORT IDataStore { InMemory = 0, OutOfCore, - Empty, - EmptyOutOfCore + Empty }; virtual ~IDataStore() = default; @@ -56,13 +55,6 @@ class SIMPLNX_EXPORT IDataStore */ virtual const ShapeType& getComponentShape() const = 0; - /** - * @brief Returns the chunk shape if the DataStore is separated into chunks. - * If the DataStore does not have chunks, this method returns a null optional. - * @return optional Shapetype - */ - virtual std::optional getChunkShape() const = 0; - /** * @brief Returns the number of values stored within the DataStore. * @return usize @@ -117,6 +109,26 @@ class SIMPLNX_EXPORT IDataStore return ""; } + /** + * @brief Returns store-specific metadata needed for crash recovery. + * + * The recovery file writer calls this to persist information about + * where this store's data lives, so it can be reattached after a + * crash. In-core stores return an empty map (their data is written + * directly to the recovery file). Out-of-core stores override this + * to return their backing file path, dataset path, chunk shape, etc. + * + * Each key-value pair is written as an HDF5 string attribute on the + * array's dataset in the recovery file. The recovery loader reads + * these attributes to reconstruct the appropriate store. + * + * @return std::map Key-value recovery metadata + */ + virtual std::map getRecoveryMetadata() const + { + return {}; + } + /** * @brief Returns the size of the stored type of the data store. * @return usize diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index da9de2fc6c..e94faafba7 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -1,9 +1,13 @@ #include "DataIOCollection.hpp" -#include "simplnx/Core/Application.hpp" +#include "simplnx/Core/Preferences.hpp" +#include "simplnx/DataStructure/AbstractStringStore.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/DataStructure/IO/Generic/CoreDataIOManager.hpp" #include "simplnx/DataStructure/IO/Generic/IDataIOManager.hpp" #include "simplnx/DataStructure/IO/HDF5/DataIOManager.hpp" +#include "simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp" +#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" namespace nx::core { @@ -21,7 +25,13 @@ void DataIOCollection::addIOManager(std::shared_ptr manager) return; } - m_ManagerMap[manager->formatName()] = manager; + const std::string& name = manager->formatName(); + if(name == Preferences::k_InMemoryFormat) + { + throw std::runtime_error(fmt::format("Cannot register an I/O manager with the reserved format name '{}'", std::string(Preferences::k_InMemoryFormat))); + } + + m_ManagerMap[name] = manager; } std::shared_ptr DataIOCollection::getManager(const std::string& formatName) const @@ -73,21 +83,132 @@ std::unique_ptr DataIOCollection::createListStore(const std::string& return coreManager.listStoreCreationFnc(coreManager.formatName())(dataType, tupleShape); } -void DataIOCollection::checkStoreDataFormat(uint64 dataSize, std::string& dataFormat) const +bool DataIOCollection::hasReadOnlyRefCreationFnc(const std::string& type) const { - if(!dataFormat.empty()) + for(const auto& [ioType, ioManager] : m_ManagerMap) { - return; + if(ioManager->hasReadOnlyRefCreationFnc(type)) + { + return true; + } + } + return false; +} + +std::unique_ptr DataIOCollection::createReadOnlyRefStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, + const ShapeType& tupleShape, const ShapeType& componentShape, const ShapeType& chunkShape) +{ + for(const auto& [ioType, ioManager] : m_ManagerMap) + { + if(ioManager->hasReadOnlyRefCreationFnc(type)) + { + auto fnc = ioManager->readOnlyRefCreationFnc(type); + return fnc(numericType, filePath, datasetPath, tupleShape, componentShape, chunkShape); + } + } + return nullptr; +} + +bool DataIOCollection::hasReadOnlyRefListCreationFnc(const std::string& type) const +{ + for(const auto& [ioType, ioManager] : m_ManagerMap) + { + if(ioManager->hasListStoreRefCreationFnc(type)) + { + return true; + } + } + return false; +} + +std::unique_ptr DataIOCollection::createReadOnlyRefListStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, + const ShapeType& tupleShape, const ShapeType& chunkShape) +{ + for(const auto& [ioType, ioManager] : m_ManagerMap) + { + if(ioManager->hasListStoreRefCreationFnc(type)) + { + auto fnc = ioManager->listStoreRefCreationFnc(type); + return fnc(numericType, filePath, datasetPath, tupleShape, chunkShape); + } + } + return nullptr; +} + +bool DataIOCollection::hasStringStoreCreationFnc(const std::string& type) const +{ + for(const auto& [ioType, ioManager] : m_ManagerMap) + { + if(ioManager->hasStringStoreCreationFnc(type)) + { + return true; + } + } + return false; +} + +std::unique_ptr DataIOCollection::createStringStore(const std::string& type, const ShapeType& tupleShape) +{ + for(const auto& [ioType, ioManager] : m_ManagerMap) + { + if(ioManager->hasStringStoreCreationFnc(type)) + { + auto fnc = ioManager->stringStoreCreationFnc(type); + return fnc(tupleShape); + } } - const Preferences* preferences = Application::GetOrCreateInstance()->getPreferences(); - const uint64 largeDataSize = preferences->valueAs(Preferences::k_LargeDataSize_Key); - const std::string largeDataFormat = preferences->valueAs(Preferences::k_PreferredLargeDataFormat_Key); - if(dataSize > largeDataSize && hasDataStoreCreationFunction(largeDataFormat)) + return nullptr; +} + +void DataIOCollection::finalizeStores(DataStructure& dataStructure) +{ + for(const auto& [ioType, ioManager] : m_ManagerMap) { - dataFormat = largeDataFormat; + if(ioManager->hasFinalizeStoresFnc(ioType)) + { + ioManager->finalizeStoresFnc(ioType)(dataStructure); + } } } +void DataIOCollection::setFormatResolver(FormatResolverFnc resolver) +{ + m_FormatResolver = std::move(resolver); +} + +bool DataIOCollection::hasFormatResolver() const +{ + return static_cast(m_FormatResolver); +} + +std::string DataIOCollection::resolveFormat(DataType numericType, const ShapeType& tupleShape, const ShapeType& componentShape, uint64 dataSizeBytes) const +{ + if(!m_FormatResolver) + { + return ""; + } + return m_FormatResolver(numericType, tupleShape, componentShape, dataSizeBytes); +} + +void DataIOCollection::setBackfillHandler(BackfillHandlerFnc handler) +{ + m_BackfillHandler = std::move(handler); +} + +bool DataIOCollection::hasBackfillHandler() const +{ + return static_cast(m_BackfillHandler); +} + +Result<> DataIOCollection::runBackfillHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader) +{ + if(!m_BackfillHandler) + { + return {}; + } + return m_BackfillHandler(dataStructure, paths, fileReader); +} + std::vector DataIOCollection::getFormatNames() const { std::vector keyNames; @@ -119,4 +240,33 @@ DataIOCollection::const_iterator DataIOCollection::end() const { return m_ManagerMap.end(); } + +void DataIOCollection::setWriteArrayOverride(WriteArrayOverrideFnc fnc) +{ + m_WriteArrayOverride = std::move(fnc); +} + +bool DataIOCollection::hasWriteArrayOverride() const +{ + return static_cast(m_WriteArrayOverride); +} + +void DataIOCollection::setWriteArrayOverrideActive(bool active) +{ + m_WriteArrayOverrideActive = active; +} + +bool DataIOCollection::isWriteArrayOverrideActive() const +{ + return m_WriteArrayOverrideActive; +} + +std::optional> DataIOCollection::runWriteArrayOverride(HDF5::DataStructureWriter& writer, const DataObject* dataObject, HDF5::GroupIO& parentGroup) +{ + if(!m_WriteArrayOverrideActive || !m_WriteArrayOverride) + { + return std::nullopt; + } + return m_WriteArrayOverride(writer, dataObject, parentGroup); +} } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index b2b46adc27..46092aded9 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -2,21 +2,37 @@ #include "simplnx/DataStructure/AbstractDataStore.hpp" #include "simplnx/DataStructure/AbstractListStore.hpp" +#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/simplnx_export.hpp" +#include "simplnx/Common/Result.hpp" #include "simplnx/Common/Types.hpp" #include "simplnx/Common/TypesUtility.hpp" +#include +#include #include #include +#include #include +#include namespace nx::core { template class AbstractDataStore; +class AbstractStringStore; +class DataObject; +class DataStructure; class IDataIOManager; +namespace HDF5 +{ +class FileIO; +class GroupIO; +class DataStructureWriter; +} // namespace HDF5 + /** * @brief The DataIOCollection class contains all known IDataIOManagers for the current Application instance. */ @@ -27,6 +43,112 @@ class SIMPLNX_EXPORT DataIOCollection using iterator = typename map_type::iterator; using const_iterator = typename map_type::const_iterator; + /** + * @brief A callback that decides the storage format for an array given its metadata. + * Returns a format string (e.g., "HDF5-OOC") or "" for default in-memory storage. + * Plugins register resolvers to control format decisions without core knowing about specific formats. + */ + using FormatResolverFnc = std::function; + + /** + * @brief A callback that takes over post-import finalization of placeholder stores. + * Called once after .dream3d paths are imported with placeholder stores (EmptyDataStore, + * EmptyListStore, EmptyStringStore). The plugin is responsible for replacing placeholders + * with real stores (e.g., OOC references, or eager-loaded data). + */ + using BackfillHandlerFnc = std::function(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader)>; + + /** + * @brief A callback that can override how a DataObject is written to HDF5. + * Used by plugins to intercept specific objects during recovery writes. + * Returns std::nullopt to pass through to the normal write path. + * Returns Result<> to indicate the hook handled the write. + */ + using WriteArrayOverrideFnc = std::function>(HDF5::DataStructureWriter& writer, const DataObject* dataObject, HDF5::GroupIO& parentGroup)>; + + /** + * @brief Sets the format resolver callback. Only one resolver can be active at a time. + * Passing nullptr disables the resolver (resolveFormat() will return ""). + * @param resolver The resolver callback (or nullptr to unset) + */ + void setFormatResolver(FormatResolverFnc resolver); + + /** + * @brief Checks whether a format resolver is registered. + * @return true if a resolver callback is currently set + */ + bool hasFormatResolver() const; + + /** + * @brief Asks the registered resolver what format to use for the given array. + * Returns "" if no resolver is registered or if the resolver decides default/in-memory. + * @param numericType The numeric type of the array + * @param tupleShape The tuple dimensions + * @param componentShape The component dimensions + * @param dataSizeBytes The total data size in bytes + * @return Format name string (e.g., "HDF5-OOC") or empty string + */ + std::string resolveFormat(DataType numericType, const ShapeType& tupleShape, const ShapeType& componentShape, uint64 dataSizeBytes) const; + + /** + * @brief Sets the backfill handler. Passing nullptr disables the handler. + * @param handler The backfill handler callback + */ + void setBackfillHandler(BackfillHandlerFnc handler); + + /** + * @brief Checks whether a backfill handler is registered. + * @return true if a handler is currently set + */ + bool hasBackfillHandler() const; + + /** + * @brief Runs the registered backfill handler. Returns an empty success Result if no handler is set. + * @param dataStructure The DataStructure containing placeholder stores + * @param paths Paths imported from the file that need finalization + * @param fileReader Open HDF5 file reader for the source file + * @return Result indicating success or failure + */ + Result<> runBackfillHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader); + + /** + * @brief Sets the write-array-override callback. Passing nullptr disables the override. + * The override is registered at plugin load time but only fires when activated + * via setWriteArrayOverrideActive(true). + * @param fnc The override callback (or nullptr to unset) + */ + void setWriteArrayOverride(WriteArrayOverrideFnc fnc); + + /** + * @brief Checks whether a write-array-override callback is registered. + * @return true if an override callback is currently set + */ + bool hasWriteArrayOverride() const; + + /** + * @brief Activates or deactivates the write-array-override. The override only + * fires when both registered AND active. Use WriteArrayOverrideGuard for RAII. + * @param active Whether the override should be active + */ + void setWriteArrayOverrideActive(bool active); + + /** + * @brief Checks whether the write-array-override is currently active. + * @return true if the override is active + */ + bool isWriteArrayOverrideActive() const; + + /** + * @brief Runs the write-array-override if registered and active. + * Returns std::nullopt if no override is registered, not active, or if the + * override passes on this object. + * @param writer The DataStructureWriter + * @param dataObject The DataObject being written + * @param parentGroup The HDF5 parent group + * @return std::optional> The override result, or std::nullopt to fall through + */ + std::optional> runWriteArrayOverride(HDF5::DataStructureWriter& writer, const DataObject* dataObject, HDF5::GroupIO& parentGroup); + DataIOCollection(); ~DataIOCollection() noexcept; @@ -109,11 +231,68 @@ class SIMPLNX_EXPORT DataIOCollection } /** - * @brief Checks and validates the data format for the given data size. - * @param dataSize The size of the data in bytes - * @param dataFormat Reference to the data format string to validate/update + * @brief Checks if any registered IO manager can create read-only reference stores. + * @param type The data format name (e.g. "HDF5-OOC") + * @return true if a read-only reference creation function is registered + */ + bool hasReadOnlyRefCreationFnc(const std::string& type) const; + + /** + * @brief Creates a read-only data store that references an existing dataset in an HDF5 file. + * @param type The data format name (e.g. "HDF5-OOC") + * @param numericType The numeric data type + * @param filePath Path to the existing HDF5 file + * @param datasetPath HDF5-internal path to the dataset + * @param tupleShape Tuple dimensions + * @param componentShape Component dimensions + * @param chunkShape Chunk dimensions for the logical cache + * @return A new read-only IDataStore, or nullptr if no factory is registered + */ + std::unique_ptr createReadOnlyRefStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, + const ShapeType& componentShape, const ShapeType& chunkShape); + + /** + * @brief Checks if any registered IO manager can create read-only reference list stores. + * @param type The format name (e.g. "HDF5-OOC") + * @return true if a registered factory exists + */ + bool hasReadOnlyRefListCreationFnc(const std::string& type) const; + + /** + * @brief Creates a read-only list store that references existing datasets in an HDF5 file. + * @param type The format name (e.g. "HDF5-OOC") + * @param numericType The numeric data type + * @param filePath Path to the existing HDF5 file + * @param datasetPath HDF5-internal path to the dataset + * @param tupleShape Tuple dimensions + * @param chunkShape Chunk dimensions for the logical cache (tuples per chunk) + * @return A new read-only IListStore, or nullptr if no factory is registered */ - void checkStoreDataFormat(uint64 dataSize, std::string& dataFormat) const; + std::unique_ptr createReadOnlyRefListStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, + const ShapeType& tupleShape, const ShapeType& chunkShape); + + /** + * @brief Checks if a string store creation function exists for the specified type. + * @param type The format name + * @return bool True if a registered factory exists + */ + bool hasStringStoreCreationFnc(const std::string& type) const; + + /** + * @brief Creates a string store of the specified type. + * @param type The format name + * @param tupleShape Tuple dimensions + * @return A new AbstractStringStore, or nullptr if no factory is registered + */ + std::unique_ptr createStringStore(const std::string& type, const ShapeType& tupleShape); + + /** + * @brief Calls all registered finalize-stores callbacks on the DataStructure. + * Called after pipeline execution to prepare stores for reading (e.g., close + * write handles and switch to read-only mode). No-op if no callbacks registered. + * @param dataStructure The DataStructure whose stores should be finalized + */ + void finalizeStores(DataStructure& dataStructure); /** * @brief Returns an iterator to the beginning of the manager collection. @@ -141,5 +320,42 @@ class SIMPLNX_EXPORT DataIOCollection private: map_type m_ManagerMap; + FormatResolverFnc m_FormatResolver; + BackfillHandlerFnc m_BackfillHandler; + WriteArrayOverrideFnc m_WriteArrayOverride; + bool m_WriteArrayOverrideActive = false; +}; + +/** + * @brief RAII guard that activates the write-array-override for its lifetime. + * On construction, sets the override active. On destruction, deactivates it. + */ +class SIMPLNX_EXPORT WriteArrayOverrideGuard +{ +public: + explicit WriteArrayOverrideGuard(std::shared_ptr ioCollection) + : m_IOCollection(std::move(ioCollection)) + { + if(m_IOCollection) + { + m_IOCollection->setWriteArrayOverrideActive(true); + } + } + + ~WriteArrayOverrideGuard() + { + if(m_IOCollection) + { + m_IOCollection->setWriteArrayOverrideActive(false); + } + } + + WriteArrayOverrideGuard(const WriteArrayOverrideGuard&) = delete; + WriteArrayOverrideGuard& operator=(const WriteArrayOverrideGuard&) = delete; + WriteArrayOverrideGuard(WriteArrayOverrideGuard&&) = delete; + WriteArrayOverrideGuard& operator=(WriteArrayOverrideGuard&&) = delete; + +private: + std::shared_ptr m_IOCollection; }; } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp index 29b80f5bcc..bc409542f5 100644 --- a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp +++ b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp @@ -1,5 +1,7 @@ #include "IDataIOManager.hpp" +#include "simplnx/DataStructure/AbstractStringStore.hpp" + namespace nx::core { IDataIOManager::IDataIOManager() = default; @@ -53,4 +55,84 @@ void IDataIOManager::addListStoreCreationFnc(const std::string& type, ListStoreC { m_ListStoreCreationMap[type] = creationFnc; } + +bool IDataIOManager::hasReadOnlyRefCreationFnc(const std::string& type) const +{ + return m_ReadOnlyRefCreationMap.find(type) != m_ReadOnlyRefCreationMap.end(); +} + +IDataIOManager::DataStoreRefCreateFnc IDataIOManager::readOnlyRefCreationFnc(const std::string& type) const +{ + auto iter = m_ReadOnlyRefCreationMap.find(type); + if(iter == m_ReadOnlyRefCreationMap.end()) + { + return nullptr; + } + return iter->second; +} + +void IDataIOManager::addReadOnlyRefCreationFnc(const std::string& type, DataStoreRefCreateFnc creationFnc) +{ + m_ReadOnlyRefCreationMap[type] = std::move(creationFnc); +} + +bool IDataIOManager::hasListStoreRefCreationFnc(const std::string& type) const +{ + return m_ListStoreRefCreationMap.find(type) != m_ListStoreRefCreationMap.cend(); +} + +IDataIOManager::ListStoreRefCreateFnc IDataIOManager::listStoreRefCreationFnc(const std::string& type) const +{ + auto iter = m_ListStoreRefCreationMap.find(type); + if(iter == m_ListStoreRefCreationMap.cend()) + { + return nullptr; + } + return iter->second; +} + +void IDataIOManager::addListStoreRefCreationFnc(const std::string& type, ListStoreRefCreateFnc creationFnc) +{ + m_ListStoreRefCreationMap[type] = std::move(creationFnc); +} + +bool IDataIOManager::hasStringStoreCreationFnc(const std::string& type) const +{ + return m_StringStoreCreationMap.find(type) != m_StringStoreCreationMap.cend(); +} + +IDataIOManager::StringStoreCreateFnc IDataIOManager::stringStoreCreationFnc(const std::string& type) const +{ + auto iter = m_StringStoreCreationMap.find(type); + if(iter == m_StringStoreCreationMap.cend()) + { + return nullptr; + } + return iter->second; +} + +void IDataIOManager::addStringStoreCreationFnc(const std::string& type, StringStoreCreateFnc creationFnc) +{ + m_StringStoreCreationMap[type] = std::move(creationFnc); +} + +bool IDataIOManager::hasFinalizeStoresFnc(const std::string& type) const +{ + return m_FinalizeStoresMap.find(type) != m_FinalizeStoresMap.end(); +} + +IDataIOManager::FinalizeStoresFnc IDataIOManager::finalizeStoresFnc(const std::string& type) const +{ + auto iter = m_FinalizeStoresMap.find(type); + if(iter == m_FinalizeStoresMap.end()) + { + return nullptr; + } + return iter->second; +} + +void IDataIOManager::addFinalizeStoresFnc(const std::string& type, FinalizeStoresFnc fnc) +{ + m_FinalizeStoresMap[type] = std::move(fnc); +} } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp index b834afc916..de0dd0d15b 100644 --- a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp +++ b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp @@ -8,6 +8,7 @@ #include "simplnx/Common/Types.hpp" +#include #include #include #include @@ -15,6 +16,8 @@ namespace nx::core { +class AbstractStringStore; +class DataStructure; class IDataFactory; /** @@ -29,8 +32,18 @@ class SIMPLNX_EXPORT IDataIOManager using factory_collection = std::map; using DataStoreCreateFnc = std::function(DataType, const ShapeType&, const ShapeType&, const std::optional&)>; using ListStoreCreateFnc = std::function(DataType, const ShapeType&)>; + using DataStoreRefCreateFnc = std::function(DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, + const ShapeType& componentShape, const ShapeType& chunkShape)>; + using ListStoreRefCreateFnc = + std::function(DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& chunkShape)>; + using StringStoreCreateFnc = std::function(const ShapeType& tupleShape)>; + using FinalizeStoresFnc = std::function; using DataStoreCreationMap = std::map; using ListStoreCreationMap = std::map; + using DataStoreRefCreationMap = std::map; + using ListStoreRefCreationMap = std::map; + using StringStoreCreationMap = std::map; + using FinalizeStoresMap = std::map; virtual ~IDataIOManager() noexcept; @@ -110,6 +123,62 @@ class SIMPLNX_EXPORT IDataIOManager */ ListStoreCreateFnc listStoreCreationFnc(const std::string& type) const; + /** + * @brief Checks if a read-only reference store creation function exists for the specified type. + * @param type The data store type name + * @return bool True if the creation function exists, false otherwise + */ + bool hasReadOnlyRefCreationFnc(const std::string& type) const; + + /** + * @brief Returns the read-only reference store creation function for the specified type. + * @param type The data store type name + * @return DataStoreRefCreateFnc The creation function, or nullptr if not found + */ + DataStoreRefCreateFnc readOnlyRefCreationFnc(const std::string& type) const; + + /** + * @brief Checks if a read-only reference list store creation function exists for the specified type. + * @param type The format name + * @return bool True if the creation function exists, false otherwise + */ + bool hasListStoreRefCreationFnc(const std::string& type) const; + + /** + * @brief Returns the read-only reference list store creation function for the specified type. + * @param type The format name + * @return ListStoreRefCreateFnc The creation function, or nullptr if not found + */ + ListStoreRefCreateFnc listStoreRefCreationFnc(const std::string& type) const; + + /** + * @brief Checks if a string store creation function exists for the specified type. + * @param type The format name + * @return bool True if the creation function exists, false otherwise + */ + bool hasStringStoreCreationFnc(const std::string& type) const; + + /** + * @brief Returns the string store creation function for the specified type. + * @param type The format name + * @return StringStoreCreateFnc The creation function, or nullptr if not found + */ + StringStoreCreateFnc stringStoreCreationFnc(const std::string& type) const; + + /** + * @brief Checks if a finalize-stores callback exists for the specified type. + * @param type The data store type name + * @return bool True if the callback exists, false otherwise + */ + bool hasFinalizeStoresFnc(const std::string& type) const; + + /** + * @brief Returns the finalize-stores callback for the specified type. + * @param type The data store type name + * @return FinalizeStoresFnc The callback, or nullptr if not found + */ + FinalizeStoresFnc finalizeStoresFnc(const std::string& type) const; + protected: /** * @brief Default constructor. @@ -130,9 +199,43 @@ class SIMPLNX_EXPORT IDataIOManager */ void addListStoreCreationFnc(const std::string& type, ListStoreCreateFnc creationFnc); + /** + * @brief Adds a read-only reference store creation function for the specified type. + * @param type The data store type name + * @param creationFnc The creation function to add + */ + void addReadOnlyRefCreationFnc(const std::string& type, DataStoreRefCreateFnc creationFnc); + + /** + * @brief Adds a read-only reference list store creation function for the specified type. + * @param type The format name + * @param creationFnc The creation function to add + */ + void addListStoreRefCreationFnc(const std::string& type, ListStoreRefCreateFnc creationFnc); + + /** + * @brief Adds a string store creation function for the specified type. + * @param type The format name + * @param creationFnc The creation function to add + */ + void addStringStoreCreationFnc(const std::string& type, StringStoreCreateFnc creationFnc); + + /** + * @brief Adds a finalize-stores callback for the specified type. + * Called after pipeline execution to prepare stores for reading (e.g., close + * write handles and switch to read-only mode). + * @param type The data store type name + * @param fnc The callback to add + */ + void addFinalizeStoresFnc(const std::string& type, FinalizeStoresFnc fnc); + private: factory_collection m_FactoryCollection; DataStoreCreationMap m_DataStoreCreationMap; ListStoreCreationMap m_ListStoreCreationMap; + DataStoreRefCreationMap m_ReadOnlyRefCreationMap; + ListStoreRefCreationMap m_ListStoreRefCreationMap; + StringStoreCreationMap m_StringStoreCreationMap; + FinalizeStoresMap m_FinalizeStoresMap; }; } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp index 908e9bdde9..d8ff27a5d1 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp @@ -1,12 +1,17 @@ #pragma once +#include "simplnx/Core/Preferences.hpp" #include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/IO/HDF5/IDataStoreIO.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.hpp" +#include #include +#include +#include + namespace nx::core { namespace HDF5 @@ -44,9 +49,19 @@ inline Result<> WriteDataStore(nx::core::HDF5::DatasetIO& datasetWriter, const A } /** - * @brief Attempts to read a DataStore from the dataset reader - * @param datasetReader - * @return std::unique_ptr> + * @brief Reads a DataStore from an HDF5 dataset. + * + * For arrays that exceed the user's largeDataSize threshold and have an OOC + * format configured (e.g. "HDF5-OOC"), this creates a read-only reference + * store pointing at the source file instead of copying data into a temp file. + * Small arrays and in-core formats load normally via CreateDataStore + readHdf5. + * + * The OOC interception is transparent: if no OOC plugin is loaded (no + * DataStoreRefCreateFnc registered), createReadOnlyRefStore returns nullptr + * and the code falls through to the normal in-core path. + * + * @param datasetReader The HDF5 dataset to read from + * @return std::shared_ptr> The created data store */ template inline std::shared_ptr> ReadDataStore(const nx::core::HDF5::DatasetIO& datasetReader) @@ -54,7 +69,130 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: auto tupleShape = IDataStoreIO::ReadTupleShape(datasetReader); auto componentShape = IDataStoreIO::ReadComponentShape(datasetReader); - // Create DataStore + // Recovery file support: detect OOC placeholder datasets that have + // backing file metadata attributes. Recovery files store OOC arrays + // as scalar placeholders with OocBackingFilePath/OocBackingDatasetPath/ + // OocChunkShape attributes. If present, reattach from the backing file + // instead of reading the placeholder data. + // No compile-time guard — createReadOnlyRefStore returns nullptr if no + // OOC plugin is loaded, providing a graceful fallback. + { + auto backingPathResult = datasetReader.readStringAttribute("OocBackingFilePath"); + if(backingPathResult.valid() && !backingPathResult.value().empty()) + { + std::filesystem::path backingFilePath = backingPathResult.value(); + + if(std::filesystem::exists(backingFilePath)) + { + auto datasetPathResult = datasetReader.readStringAttribute("OocBackingDatasetPath"); + std::string backingDatasetPath = datasetPathResult.valid() ? datasetPathResult.value() : ""; + + // Parse chunk shape from comma-separated string (e.g. "1,200,200") + auto chunkResult = datasetReader.readStringAttribute("OocChunkShape"); + ShapeType chunkShape; + if(chunkResult.valid() && !chunkResult.value().empty()) + { + std::stringstream ss(chunkResult.value()); + std::string token; + while(std::getline(ss, token, ',')) + { + if(!token.empty()) + { + chunkShape.push_back(static_cast(std::stoull(token))); + } + } + } + + auto ioCollection = DataStoreUtilities::GetIOCollection(); + auto refStore = ioCollection->createReadOnlyRefStore("HDF5-OOC", GetDataType(), backingFilePath, backingDatasetPath, tupleShape, componentShape, chunkShape); + if(refStore != nullptr) + { + return std::shared_ptr>(std::dynamic_pointer_cast>(std::shared_ptr(std::move(refStore)))); + } + // If createReadOnlyRefStore returns nullptr, fall through to normal path + } + else + { + std::cerr << "[RECOVERY] Backing file missing for OOC array: " << backingFilePath << std::endl; + // Fall through to load placeholder data + } + } + } + + // Determine what format this array should use based on its byte size + // and the user's OOC preferences (largeDataSize threshold, force flag). + // The plugin-registered resolver encapsulates all format-selection policy. + uint64 dataSize = DataStoreUtilities::CalculateDataSize(tupleShape, componentShape); + auto ioCollection = DataStoreUtilities::GetIOCollection(); + std::string dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, componentShape, dataSize); + + // If the format is an OOC format (not empty, not explicit in-core), and + // we're reading from an existing HDF5 file, create a read-only reference + // store pointing at the source file. This avoids copying gigabytes of + // data into a temporary file for arrays that already exist on disk. + if(!dataFormat.empty() && dataFormat != Preferences::k_InMemoryFormat) + { + auto filePath = datasetReader.getFilePath(); + // ObjectIO::getFilePath() may return empty when the DatasetIO was + // created via the parentId constructor (GroupIO::openDataset). + // Fall back to querying HDF5 directly for the file name. + if(filePath.empty()) + { + hid_t objId = datasetReader.getId(); + if(objId > 0) + { + ssize_t nameLen = H5Fget_name(objId, nullptr, 0); + if(nameLen > 0) + { + std::string nameBuf(static_cast(nameLen), '\0'); + H5Fget_name(objId, nameBuf.data(), static_cast(nameLen) + 1); + filePath = nameBuf; + } + } + } + auto datasetPath = datasetReader.getObjectPath(); + auto chunkDims = datasetReader.getChunkDimensions(); + + // Compute chunk shape in tuple space. HDF5 chunk dims include both + // tuple and component dimensions; we take only the tuple portion. + // For contiguous datasets (empty chunkDims), use a default shape + // optimized for Z-slice-sequential access. + ShapeType chunkShape; + if(!chunkDims.empty() && chunkDims.size() >= tupleShape.size()) + { + chunkShape.assign(chunkDims.begin(), chunkDims.begin() + static_cast(tupleShape.size())); + } + else if(tupleShape.size() >= 3) + { + // 3D+ contiguous: {1, Y, X, ...} — one Z-slice per logical chunk + chunkShape.resize(tupleShape.size()); + chunkShape[0] = 1; + for(usize d = 1; d < tupleShape.size(); ++d) + { + chunkShape[d] = tupleShape[d]; + } + } + else + { + // 1D/2D contiguous: clamp each dimension to 64 + chunkShape.resize(tupleShape.size()); + for(usize d = 0; d < tupleShape.size(); ++d) + { + chunkShape[d] = std::min(static_cast(64), tupleShape[d]); + } + } + + // createReadOnlyRefStore returns nullptr if no ref factory is registered + // (e.g. SimplnxOoc plugin not loaded). In that case, fall through to + // the normal in-core path below. + auto refStore = ioCollection->createReadOnlyRefStore(dataFormat, GetDataType(), filePath, datasetPath, tupleShape, componentShape, chunkShape); + if(refStore != nullptr) + { + return std::shared_ptr>(std::dynamic_pointer_cast>(std::shared_ptr(std::move(refStore)))); + } + } + + // Normal path: create an in-core DataStore and read the data from HDF5. auto dataStore = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); dataStore->readHdf5(datasetReader); return dataStore; diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp index cd4f4ae3ed..462fdd6bf9 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp @@ -2,8 +2,10 @@ #include "simplnx/Core/Application.hpp" #include "simplnx/DataStructure/INeighborList.hpp" +#include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" #include "simplnx/DataStructure/IO/HDF5/DataIOManager.hpp" #include "simplnx/DataStructure/IO/HDF5/IDataIO.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -115,21 +117,32 @@ Result<> DataStructureWriter::writeDataObject(const DataObject* dataObject, nx:: // Create an HDF5 link return writeDataObjectLink(dataObject, parentGroup); } - else + + // Write-array-override hook: if active, give the registered plugin a chance + // to handle this object (e.g., writing a recovery placeholder for OOC arrays). + // If the hook returns std::nullopt, fall through to the normal write path. + auto ioCollection = DataStoreUtilities::GetIOCollection(); + if(ioCollection->isWriteArrayOverrideActive()) { - // Write new data - auto factory = m_IOManager->getFactoryAs(dataObject->getTypeName()); - if(factory == nullptr) + auto overrideResult = ioCollection->runWriteArrayOverride(*this, dataObject, parentGroup); + if(overrideResult.has_value()) { - std::string ss = fmt::format("Could not find IO factory for datatype: {}", dataObject->getTypeName()); - return MakeErrorResult(-5, ss); + return overrideResult.value(); } + } - auto result = factory->writeDataObject(*this, dataObject, parentGroup); - if(result.invalid()) - { - return result; - } + // Normal write path + auto factory = m_IOManager->getFactoryAs(dataObject->getTypeName()); + if(factory == nullptr) + { + std::string ss = fmt::format("Could not find IO factory for datatype: {}", dataObject->getTypeName()); + return MakeErrorResult(-5, ss); + } + + auto result = factory->writeDataObject(*this, dataObject, parentGroup); + if(result.invalid()) + { + return result; } return {}; diff --git a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp index 6475ce80cb..d306a4b806 100644 --- a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp @@ -9,7 +9,12 @@ #include "simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp" #include "simplnx/DataStructure/IO/HDF5/IDataIO.hpp" #include "simplnx/DataStructure/NeighborList.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" +#include + +#include +#include #include namespace nx::core @@ -151,6 +156,71 @@ class NeighborListIO : public IDataIO auto numNeighborsPtr = DataStoreIO::ReadDataStore(numNeighborsReader); auto& numNeighborsStore = *numNeighborsPtr.get(); + // Compute total neighbors and tuple shape before deciding in-memory vs OOC. + // The total number of elements in the flat dataset is the sum of all NumNeighbors values. + const auto numTuples = numNeighborsStore.getNumberOfTuples(); + const auto tupleShape = numNeighborsStore.getTupleShape(); + uint64 totalNeighbors = 0; + for(usize i = 0; i < numTuples; i++) + { + totalNeighbors += static_cast(numNeighborsStore[i]); + } + + // Query the format resolver for OOC-vs-in-memory decision. The resolver is + // plugin-registered (SimplnxOoc); if no plugin is loaded, the format is empty + // and we fall through to in-memory loading. Also check that a read-only list + // store factory is registered for the format; if not, fall through. + auto ioCollection = DataStoreUtilities::GetIOCollection(); + const uint64 dataSize = totalNeighbors * sizeof(T); + const std::string dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, {1}, dataSize); + + if(!dataFormat.empty() && ioCollection->hasReadOnlyRefListCreationFnc(dataFormat)) + { + // Resolve the backing file path. GroupIO::getFilePath() may return empty when + // the GroupIO was created via a parentId constructor; fall back to HDF5's + // H5Fget_name using the dataset's id. + std::filesystem::path filePath = parentGroup.getFilePath(); + if(filePath.empty()) + { + hid_t objId = dataReader.getId(); + if(objId > 0) + { + ssize_t nameLen = H5Fget_name(objId, nullptr, 0); + if(nameLen > 0) + { + std::string nameBuf(static_cast(nameLen), '\0'); + H5Fget_name(objId, nameBuf.data(), static_cast(nameLen) + 1); + filePath = nameBuf; + } + } + } + + // HDF5-internal dataset path for the flat neighbor data array. + const std::string datasetPath = dataReader.getObjectPath(); + + // Chunk tuple count: for 3D+ tuple shapes use a single Z-slice (Y*X), + // otherwise chunk over the total tuple count. + const usize chunkTupleCount = (tupleShape.size() >= 3) ? (tupleShape[tupleShape.size() - 2] * tupleShape[tupleShape.size() - 1]) : + std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>()); + const ShapeType chunkShape = {chunkTupleCount}; + + auto ocStore = ioCollection->createReadOnlyRefListStore(dataFormat, GetDataType(), filePath, datasetPath, tupleShape, chunkShape); + if(ocStore != nullptr) + { + auto* rawStore = ocStore.release(); + auto* typedStore = dynamic_cast*>(rawStore); + if(typedStore != nullptr) + { + auto sharedStore = std::shared_ptr>(typedStore); + neighborList.setStore(sharedStore); + return {}; + } + // dynamic_cast failed, clean up and fall through to in-memory path + delete rawStore; + } + } + + // In-memory path: read the flat data and scatter into per-tuple lists. auto flatDataStorePtr = dataReader.template readAsDataStore(); if(flatDataStorePtr == nullptr) { @@ -163,8 +233,7 @@ class NeighborListIO : public IDataIO } usize offset = 0; - const auto numTuples = numNeighborsStore.getNumberOfTuples(); - auto listStorePtr = DataStoreUtilities::CreateListStore(numNeighborsStore.getTupleShape()); + auto listStorePtr = DataStoreUtilities::CreateListStore(tupleShape); AbstractListStore& listStore = *listStorePtr.get(); for(usize i = 0; i < numTuples; i++) { diff --git a/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp b/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp index 7a5116aabe..9eeaf3feb9 100644 --- a/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp +++ b/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp @@ -1,7 +1,7 @@ #include "StringArrayIO.hpp" -#include #include "DataStructureReader.hpp" +#include "simplnx/DataStructure/EmptyStringStore.hpp" #include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/DataStructure/StringStore.hpp" @@ -52,10 +52,25 @@ Result<> StringArrayIO::readData(DataStructureReader& dataStructureReader, const { tupleShape = std::move(tupleShapeResult.value()); } - usize numValues = std::accumulate(tupleShape.cbegin(), tupleShape.cend(), 1ULL, std::multiplies<>()); - std::vector strings = useEmptyDataStore ? std::vector(numValues) : datasetReader.readAsVectorOfStrings(); - const auto* data = StringArray::Import(dataStructureReader.getDataStructure(), dataArrayName, tupleShape, importId, std::move(strings), parentId); + StringArray* data = nullptr; + if(useEmptyDataStore) + { + // Import with an empty vector to avoid allocating numValues empty strings, then swap in + // an EmptyStringStore placeholder. The actual string data will be loaded later by the + // backfill handler. + data = StringArray::Import(dataStructureReader.getDataStructure(), dataArrayName, tupleShape, importId, std::vector{}, parentId); + if(data != nullptr) + { + auto emptyStore = std::make_shared(tupleShape); + data->setStore(emptyStore); + } + } + else + { + std::vector strings = datasetReader.readAsVectorOfStrings(); + data = StringArray::Import(dataStructureReader.getDataStructure(), dataArrayName, tupleShape, importId, std::move(strings), parentId); + } if(data == nullptr) { diff --git a/src/simplnx/DataStructure/StringStore.hpp b/src/simplnx/DataStructure/StringStore.hpp index b6eaa39480..d6560354ed 100644 --- a/src/simplnx/DataStructure/StringStore.hpp +++ b/src/simplnx/DataStructure/StringStore.hpp @@ -7,7 +7,20 @@ namespace nx::core { -class StringStore : public AbstractStringStore +/** + * @class StringStore + * @brief The concrete in-memory string storage backend for StringArray. + * + * StringStore owns a `std::vector` and provides full + * read/write access to its elements. This is the "real" store that holds + * loaded string data, as opposed to EmptyStringStore which is a + * metadata-only placeholder. + * + * @see AbstractStringStore The abstract interface this class implements. + * @see EmptyStringStore The placeholder counterpart used during preflight + * or OOC skeleton construction. + */ +class SIMPLNX_EXPORT StringStore : public AbstractStringStore { public: /** @@ -26,7 +39,7 @@ class StringStore : public AbstractStringStore /** * @brief Destructor. */ - ~StringStore(); + ~StringStore() override; /** * @brief Creates a deep copy of this StringStore. @@ -65,6 +78,21 @@ class StringStore : public AbstractStringStore */ bool empty() const override; + /** + * @brief Returns false because StringStore always contains real, accessible + * string data (backed by a std::vector). + * + * This distinguishes StringStore from EmptyStringStore, which is a + * metadata-only placeholder. Import/backfill code uses isPlaceholder() + * to decide which string arrays still need their data loaded from disk. + * + * @return false Always returns false for StringStore. + */ + bool isPlaceholder() const override + { + return false; + } + /** * @brief Array subscript operator to access the string at the specified index. * @param index The index to access diff --git a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp index 8e89250d0e..58731528d5 100644 --- a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp +++ b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp @@ -1,8 +1,6 @@ #include "ImportH5ObjectPathsAction.hpp" -#include "simplnx/DataStructure/BaseGroup.hpp" -#include "simplnx/DataStructure/DataArray.hpp" -#include "simplnx/DataStructure/DataStore.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -50,6 +48,15 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod importStructure.resetIds(dataStructure.getNextId()); const bool preflighting = mode == Mode::Preflight; + + // Generic: "does a plugin want to defer loading and take over finalization?" + bool useDeferredLoad = false; + if(!preflighting) + { + auto ioCollection = DataStoreUtilities::GetIOCollection(); + useDeferredLoad = ioCollection->hasBackfillHandler(); + } + std::stringstream errorMessages; for(const auto& targetPath : m_Paths) { @@ -58,7 +65,10 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod return MakeErrorResult(-6203, fmt::format("{}Unable to import DataObject at '{}' because an object already exists there. Consider a rename of existing object.", prefix, targetPath.toString())); } - auto result = DREAM3D::FinishImportingObject(importStructure, dataStructure, targetPath, fileReader, preflighting); + // When deferring loading, pass preflight=true to avoid the expensive + // data copy. The resulting placeholder stores are finalized by the + // registered backfill handler below. + auto result = DREAM3D::FinishImportingObject(importStructure, dataStructure, targetPath, fileReader, preflighting || useDeferredLoad); if(result.invalid()) { for(const auto& errorResult : result.errors()) @@ -72,6 +82,17 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod return MakeErrorResult(-6201, errorMessages.str()); } + // Delegate placeholder finalization to the registered backfill handler (if any) + if(useDeferredLoad) + { + auto ioCollection = DataStoreUtilities::GetIOCollection(); + auto backfillResult = ioCollection->runBackfillHandler(dataStructure, m_Paths, fileReader); + if(backfillResult.invalid()) + { + return backfillResult; + } + } + return ConvertResult(std::move(dataStructureResult)); } diff --git a/src/simplnx/Parameters/ArraySelectionParameter.cpp b/src/simplnx/Parameters/ArraySelectionParameter.cpp index af1ba3ff33..b8635a4c0c 100644 --- a/src/simplnx/Parameters/ArraySelectionParameter.cpp +++ b/src/simplnx/Parameters/ArraySelectionParameter.cpp @@ -201,11 +201,7 @@ Result<> ArraySelectionParameter::validatePath(const DataStructure& dataStructur { IDataStore::StoreType storeType = dataArray->getStoreType(); - if(allowsInMemory() && (storeType == IDataStore::StoreType::Empty)) - { - return {}; - } - else if(allowsOutOfCore() && (storeType == IDataStore::StoreType::EmptyOutOfCore)) + if(storeType == IDataStore::StoreType::Empty) { return {}; } diff --git a/src/simplnx/Utilities/AlgorithmDispatch.hpp b/src/simplnx/Utilities/AlgorithmDispatch.hpp index 564d277c41..7735f5bf16 100644 --- a/src/simplnx/Utilities/AlgorithmDispatch.hpp +++ b/src/simplnx/Utilities/AlgorithmDispatch.hpp @@ -2,7 +2,9 @@ #include "simplnx/Common/Result.hpp" #include "simplnx/DataStructure/IDataArray.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" +#include #include namespace nx::core @@ -11,7 +13,7 @@ namespace nx::core /** * @brief Checks whether an IDataArray is backed by out-of-core (chunked) storage. * - * Returns true when the array's data store reports a chunk shape (e.g. ZarrStore), + * Returns true when the array's data store reports StoreType::OutOfCore, * indicating that data lives on disk in compressed chunks rather than in a * contiguous in-memory buffer. * @@ -20,7 +22,7 @@ namespace nx::core */ inline bool IsOutOfCore(const IDataArray& array) { - return array.getIDataStoreRef().getChunkShape().has_value(); + return array.getIDataStoreRef().getStoreType() == IDataStore::StoreType::OutOfCore; } /** @@ -60,15 +62,48 @@ inline bool& ForceOocAlgorithm() return s_force; } +/** + * @brief Integer array of forceOoc values for Catch2 GENERATE(from_range(...)). + * + * Controlled by CMake option SIMPLNX_TEST_ALGORITHM_PATH (passed as a + * compile definition to test targets): + * 0 (Both) - {0, 1}: tests both in-core and OOC paths (default) + * 1 (OocOnly) - {1}: tests only OOC path (use for OOC builds) + * 2 (InCoreOnly) - {0}: tests only in-core path (quick validation) + * + * Uses int instead of bool because Catch2 v2's FixedValuesGenerator + * does not support bool due to std::vector specialization. + * + * Usage in tests: + * @code + * bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + * const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + * @endcode + * + * Set via: cmake -DSIMPLNX_TEST_ALGORITHM_PATH=1 ... + */ +#ifndef SIMPLNX_TEST_ALGORITHM_PATH +#define SIMPLNX_TEST_ALGORITHM_PATH 0 +#endif + +// clang-format off +#if SIMPLNX_TEST_ALGORITHM_PATH == 1 +inline const std::array k_ForceOocTestValues = {1}; +#elif SIMPLNX_TEST_ALGORITHM_PATH == 2 +inline const std::array k_ForceOocTestValues = {0}; +#else +inline const std::array k_ForceOocTestValues = {0, 1}; +#endif +// clang-format on + /** * @brief RAII guard that sets ForceOocAlgorithm() on construction and * restores the previous value on destruction. * - * Usage in tests with Catch2 GENERATE: + * Usage in tests: * @code - * bool forceOoc = GENERATE(false, true); + * bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); * const nx::core::ForceOocAlgorithmGuard guard(forceOoc); - * // ... test body runs with both algorithm paths ... * @endcode */ class ForceOocAlgorithmGuard @@ -91,7 +126,61 @@ class ForceOocAlgorithmGuard ForceOocAlgorithmGuard& operator=(ForceOocAlgorithmGuard&&) = delete; private: - bool m_Original; + bool m_Original = false; +}; + +/** + * @brief Returns a reference to the global flag that forces DispatchAlgorithm + * to always select the in-core algorithm, overriding storage-type detection. + * + * This is primarily used in unit tests to exercise the in-core algorithm path + * even when data is stored out-of-core (e.g., loaded from HDF5 in an OOC build). + * Use ForceInCoreAlgorithmGuard for RAII-safe toggling in tests. + * + * ForceInCoreAlgorithm() takes precedence over ForceOocAlgorithm() and + * AnyOutOfCore() in DispatchAlgorithm. + * + * @return Reference to the static force flag + */ +inline bool& ForceInCoreAlgorithm() +{ + static bool s_force = false; + return s_force; +} + +/** + * @brief RAII guard that sets ForceInCoreAlgorithm() on construction and + * restores the previous value on destruction. + * + * Use this in tests that verify the in-core algorithm path when data may be + * loaded from disk as OOC (e.g., dream3d files in an OOC-enabled build). + * + * Usage in tests: + * @code + * const nx::core::ForceInCoreAlgorithmGuard guard; + * @endcode + */ +class ForceInCoreAlgorithmGuard +{ +public: + ForceInCoreAlgorithmGuard() + : m_Original(ForceInCoreAlgorithm()) + { + ForceInCoreAlgorithm() = true; + } + + ~ForceInCoreAlgorithmGuard() + { + ForceInCoreAlgorithm() = m_Original; + } + + ForceInCoreAlgorithmGuard(const ForceInCoreAlgorithmGuard&) = delete; + ForceInCoreAlgorithmGuard(ForceInCoreAlgorithmGuard&&) = delete; + ForceInCoreAlgorithmGuard& operator=(const ForceInCoreAlgorithmGuard&) = delete; + ForceInCoreAlgorithmGuard& operator=(ForceInCoreAlgorithmGuard&&) = delete; + +private: + bool m_Original = false; }; /** @@ -105,12 +194,11 @@ class ForceOocAlgorithmGuard * a different algorithm (e.g. scanline CCL with sequential chunk access) can be * orders of magnitude faster for OOC data. * - * This function checks the storage type of the given arrays and the global force - * flag. If *any* array is out-of-core or ForceOocAlgorithm() is true, the OOC - * algorithm is selected. Callers should pass all input and output arrays the - * filter operates on. Both algorithm classes must: - * - Be constructible from the same ArgsT... parameter pack - * - Provide operator()() returning Result<> + * Selection logic (evaluated in order): + * 1. ForceInCoreAlgorithm() == true -> always use InCoreAlgo + * 2. AnyOutOfCore(arrays) == true -> use OocAlgo + * 3. ForceOocAlgorithm() == true -> use OocAlgo + * 4. Otherwise -> use InCoreAlgo * * @tparam InCoreAlgo Algorithm class optimized for in-memory data * @tparam OocAlgo Algorithm class optimized for out-of-core (chunked) data @@ -122,7 +210,7 @@ class ForceOocAlgorithmGuard template Result<> DispatchAlgorithm(std::initializer_list arrays, ArgsT&&... args) { - if(AnyOutOfCore(arrays) || ForceOocAlgorithm()) + if(!ForceInCoreAlgorithm() && (AnyOutOfCore(arrays) || ForceOocAlgorithm())) { return OocAlgo(std::forward(args)...)(); } diff --git a/src/simplnx/Utilities/ArrayCreationUtilities.cpp b/src/simplnx/Utilities/ArrayCreationUtilities.cpp index ce74100bbc..1476106c44 100644 --- a/src/simplnx/Utilities/ArrayCreationUtilities.cpp +++ b/src/simplnx/Utilities/ArrayCreationUtilities.cpp @@ -10,8 +10,8 @@ bool ArrayCreationUtilities::CheckMemoryRequirement(DataStructure& dataStructure { static const uint64 k_AvailableMemory = Memory::GetTotalMemory(); - // Only check if format is set to in-memory - if(!format.empty()) + // If a format is already specified (and it's not the in-memory sentinel), skip the check. + if(!format.empty() && format != Preferences::k_InMemoryFormat) { return true; } @@ -22,15 +22,19 @@ bool ArrayCreationUtilities::CheckMemoryRequirement(DataStructure& dataStructure const uint64 largeDataStructureSize = preferencesPtr->largeDataStructureSize(); const std::string largeDataFormat = preferencesPtr->largeDataFormat(); + // Check if OOC is available: format must be non-empty and not the in-memory sentinel. + bool oocAvailable = !largeDataFormat.empty() && largeDataFormat != Preferences::k_InMemoryFormat; + if(memoryUsage >= largeDataStructureSize) { - // Check if out-of-core is available / enabled - if(largeDataFormat.empty() && memoryUsage >= k_AvailableMemory) + if(!oocAvailable && memoryUsage >= k_AvailableMemory) { return false; } - // Use out-of-core - format = largeDataFormat; + if(oocAvailable) + { + format = largeDataFormat; + } } return true; diff --git a/src/simplnx/Utilities/DataStoreUtilities.cpp b/src/simplnx/Utilities/DataStoreUtilities.cpp index 55edfeb3d5..1c25beda9a 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.cpp +++ b/src/simplnx/Utilities/DataStoreUtilities.cpp @@ -4,16 +4,6 @@ using namespace nx::core; -//----------------------------------------------------------------------------- -void DataStoreUtilities::TryForceLargeDataFormatFromPrefs(std::string& dataFormat) -{ - auto* preferencesPtr = Application::GetOrCreateInstance()->getPreferences(); - if(preferencesPtr->forceOocData()) - { - dataFormat = preferencesPtr->largeDataFormat(); - } -} - //----------------------------------------------------------------------------- std::shared_ptr DataStoreUtilities::GetIOCollection() { diff --git a/src/simplnx/Utilities/DataStoreUtilities.hpp b/src/simplnx/Utilities/DataStoreUtilities.hpp index aa6ae83f90..f34b048f67 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.hpp +++ b/src/simplnx/Utilities/DataStoreUtilities.hpp @@ -12,13 +12,6 @@ namespace nx::core::DataStoreUtilities { -/** - * @brief Sets the dataFormat string to the large data format from the prefs - * if forceOocData is true. - * @param dataFormat - */ -SIMPLNX_EXPORT void TryForceLargeDataFormatFromPrefs(std::string& dataFormat); - /** * @brief Returns the application's DataIOCollection. * @return @@ -51,9 +44,11 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap } case IDataAction::Mode::Execute: { uint64 dataSize = CalculateDataSize(tupleShape, componentShape); - TryForceLargeDataFormatFromPrefs(dataFormat); auto ioCollection = GetIOCollection(); - ioCollection->checkStoreDataFormat(dataSize, dataFormat); + if(dataFormat.empty()) + { + dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, componentShape, dataSize); + } return ioCollection->createDataStoreWithType(dataFormat, tupleShape, componentShape); } default: { @@ -72,9 +67,11 @@ std::shared_ptr> CreateListStore(const ShapeType& tupleShap } case IDataAction::Mode::Execute: { uint64 dataSize = CalculateDataSize(tupleShape, {10}); - TryForceLargeDataFormatFromPrefs(dataFormat); auto ioCollection = GetIOCollection(); - ioCollection->checkStoreDataFormat(dataSize, dataFormat); + if(dataFormat.empty()) + { + dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, {10}, dataSize); + } return ioCollection->createListStoreWithType(dataFormat, tupleShape); } default: { diff --git a/src/simplnx/Utilities/IParallelAlgorithm.cpp b/src/simplnx/Utilities/IParallelAlgorithm.cpp index dc04752a06..6b6b22a7f2 100644 --- a/src/simplnx/Utilities/IParallelAlgorithm.cpp +++ b/src/simplnx/Utilities/IParallelAlgorithm.cpp @@ -1,6 +1,6 @@ #include "IParallelAlgorithm.hpp" -#include "simplnx/Core/Application.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" namespace { @@ -19,7 +19,7 @@ bool CheckStoresInMemory(const nx::core::IParallelAlgorithm::AlgorithmStores& st continue; } - if(!storePtr->getDataFormat().empty()) + if(storePtr->getStoreType() == nx::core::IDataStore::StoreType::OutOfCore) { return false; } @@ -43,7 +43,7 @@ bool CheckArraysInMemory(const nx::core::IParallelAlgorithm::AlgorithmArrays& ar continue; } - if(!arrayPtr->getIDataStoreRef().getDataFormat().empty()) + if(arrayPtr->getIDataStoreRef().getStoreType() == nx::core::IDataStore::StoreType::OutOfCore) { return false; } @@ -58,10 +58,11 @@ namespace nx::core // ----------------------------------------------------------------------------- IParallelAlgorithm::IParallelAlgorithm() { -#ifdef SIMPLNX_ENABLE_MULTICORE - // Do not run OOC data in parallel by default. - m_RunParallel = !Application::GetOrCreateInstance()->getPreferences()->useOocData(); -#endif + // m_RunParallel defaults to true (ifdef SIMPLNX_ENABLE_MULTICORE) or false. + // Individual filters disable via requireArraysInMemory()/requireStoresInMemory() + // if they genuinely need in-memory data (e.g., ITK filters). + // OOC stores are now thread-safe (ChunkCache + HDF5 global mutex), so + // TBB parallelism is safe on OOC data. } // ----------------------------------------------------------------------------- diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp index 9a79726a53..e25e07378b 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp @@ -15,6 +15,7 @@ #include "simplnx/DataStructure/Geometry/TetrahedralGeom.hpp" #include "simplnx/DataStructure/Geometry/TriangleGeom.hpp" #include "simplnx/DataStructure/Geometry/VertexGeom.hpp" +#include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" #include "simplnx/DataStructure/IO/HDF5/DataStructureReader.hpp" #include "simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp" #include "simplnx/DataStructure/IO/HDF5/IDataStoreIO.hpp" @@ -23,6 +24,7 @@ #include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/DataStructure/StringStore.hpp" #include "simplnx/Pipeline/Pipeline.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" #include @@ -1096,9 +1098,9 @@ Result readLegacyNodeConnectivityList(DataStructure& dataStructure template Result<> createLegacyNeighborList(DataStructure& dataStructure, DataObject ::IdType parentId, const nx::core::HDF5::GroupIO& parentReader, const nx::core::HDF5::DatasetIO& datasetReader, - const ShapeType& tupleDims) + const ShapeType& tupleDims, bool preflight = false) { - auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader); + auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader, preflight); auto* neighborList = NeighborList::Create(dataStructure, datasetReader.getName(), listStore, parentId); if(neighborList == nullptr) { @@ -1108,7 +1110,8 @@ Result<> createLegacyNeighborList(DataStructure& dataStructure, DataObject ::IdT return {}; } -Result<> readLegacyNeighborList(DataStructure& dataStructure, const nx::core::HDF5::GroupIO& parentReader, const nx::core::HDF5::DatasetIO& datasetReader, DataObject::IdType parentId) +Result<> readLegacyNeighborList(DataStructure& dataStructure, const nx::core::HDF5::GroupIO& parentReader, const nx::core::HDF5::DatasetIO& datasetReader, DataObject::IdType parentId, + bool preflight = false) { auto dataTypeResult = datasetReader.getDataType(); if(dataTypeResult.invalid()) @@ -1126,36 +1129,36 @@ Result<> readLegacyNeighborList(DataStructure& dataStructure, const nx::core::HD switch(dataType) { case DataType::float32: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::float64: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::boolean: [[fallthrough]]; case DataType::int8: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::int16: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::int32: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::int64: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::uint8: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::uint16: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::uint32: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; case DataType::uint64: - result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims); + result = createLegacyNeighborList(dataStructure, parentId, parentReader, datasetReader, tDims, preflight); break; } @@ -1252,7 +1255,7 @@ Result<> readLegacyArray(DataStructure& dataStructure, const nx::core::HDF5::Gro auto dataArraySet = amGroupReader.openDataset(arrayName); if(isLegacyNeighborList(dataArraySet)) { - return readLegacyNeighborList(dataStructure, amGroupReader, dataArraySet, 0); + return readLegacyNeighborList(dataStructure, amGroupReader, dataArraySet, 0, preflight); } else if(isLegacyStringArray(dataArraySet)) { @@ -1614,7 +1617,7 @@ Result<> readLegacyAttributeMatrix(DataStructure& dataStructure, const nx::core: if(isLegacyNeighborList(dataArraySet)) { - daResults.push_back(readLegacyNeighborList(dataStructure, amGroupReader, dataArraySet, attributeMatrix->getId())); + daResults.push_back(readLegacyNeighborList(dataStructure, amGroupReader, dataArraySet, attributeMatrix->getId(), preflight)); } else if(isLegacyStringArray(dataArraySet)) { @@ -2456,6 +2459,13 @@ Result<> DREAM3D::WriteFile(const std::filesystem::path& path, const DataStructu return {}; } +Result<> DREAM3D::WriteRecoveryFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline) +{ + auto ioCollection = DataStoreUtilities::GetIOCollection(); + WriteArrayOverrideGuard guard(ioCollection); + return WriteFile(path, dataStructure, pipeline, false); +} + Result<> DREAM3D::AppendFile(const std::filesystem::path& path, const DataStructure& dataStructure, const DataPath& dataPath) { auto file = nx::core::HDF5::FileIO::AppendFile(path); diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp index 7ca661751c..e88bbd48b8 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp @@ -99,6 +99,19 @@ SIMPLNX_EXPORT Result<> WriteFile(nx::core::HDF5::FileIO& fileWriter, const Pipe */ SIMPLNX_EXPORT Result<> WriteFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline = {}, bool writeXdmf = false); +/** + * @brief Writes a recovery .dream3d file. Activates the write-array-override + * hook so that plugins (e.g., SimplnxOoc) can write lightweight placeholders + * for OOC arrays instead of materializing full data. In-core arrays are + * written normally via the standard write path. If no override is registered, + * behaves identically to WriteFile. + * @param path Output file path + * @param dataStructure The DataStructure to write + * @param pipeline The Pipeline to write + * @return Result<> with any errors + */ +SIMPLNX_EXPORT Result<> WriteRecoveryFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline = {}); + /** * @brief Appends the object at the path in the data structure to the dream3d file * @param path diff --git a/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp b/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp index 1366a2188e..dbf9617c7e 100644 --- a/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp +++ b/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp @@ -2,6 +2,8 @@ #include "H5Support.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" + namespace nx::core::HDF5 { namespace Support @@ -80,7 +82,7 @@ Result<> FillDataArray(DataStructure& dataStructure, const DataPath& dataArrayPa const std::optional>& count = std::nullopt) { auto& dataArray = dataStructure.getDataRefAs>(dataArrayPath); - if(dataArray.getDataFormat().empty()) + if(dataArray.getIDataStoreRef().getStoreType() != IDataStore::StoreType::OutOfCore) { return FillDataStore(dataArray, dataArrayPath, datasetReader, start, count); } diff --git a/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp b/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp index 988519b9fe..fa0021ab96 100644 --- a/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp +++ b/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp @@ -961,6 +961,89 @@ Result<> DatasetIO::writeSpan(const DimsType& dims, nonstd::span +Result<> DatasetIO::createEmptyDataset(const DimsType& dims) +{ + hid_t dataType = HdfTypeForPrimitive(); + if(dataType == -1) + { + return MakeErrorResult(-1020, "createEmptyDataset error: Unsupported data type."); + } + + std::vector hDims(dims.size()); + std::transform(dims.begin(), dims.end(), hDims.begin(), [](DimsType::value_type x) { return static_cast(x); }); + hid_t dataspaceId = H5Screate_simple(static_cast(hDims.size()), hDims.data(), nullptr); + if(dataspaceId < 0) + { + return MakeErrorResult(-1021, "createEmptyDataset error: Unable to create dataspace."); + } + + auto datasetId = createOrOpenDataset(dataspaceId); + H5Sclose(dataspaceId); + if(datasetId < 0) + { + return MakeErrorResult(-1022, "createEmptyDataset error: Unable to create dataset."); + } + + return {}; +} + +template +Result<> DatasetIO::writeSpanHyperslab(nonstd::span values, const std::vector& start, const std::vector& count) +{ + if(!isValid()) + { + return MakeErrorResult(-506, fmt::format("Cannot open HDF5 data at {} / {}", getFilePath().string(), getNamePath())); + } + + hid_t dataType = HdfTypeForPrimitive(); + if(dataType == -1) + { + return MakeErrorResult(-1010, "writeSpanHyperslab error: Unsupported data type."); + } + + hid_t datasetId = open(); + hid_t fileSpaceId = H5Dget_space(datasetId); + if(fileSpaceId < 0) + { + return MakeErrorResult(-1011, "writeSpanHyperslab error: Unable to open the dataspace."); + } + + // Select hyperslab in the file dataspace +#if defined(__APPLE__) + std::vector startVec(start.begin(), start.end()); + std::vector countVec(count.begin(), count.end()); + if(H5Sselect_hyperslab(fileSpaceId, H5S_SELECT_SET, startVec.data(), NULL, countVec.data(), NULL) < 0) +#else + if(H5Sselect_hyperslab(fileSpaceId, H5S_SELECT_SET, start.data(), NULL, count.data(), NULL) < 0) +#endif + { + H5Sclose(fileSpaceId); + return MakeErrorResult(-1012, "writeSpanHyperslab error: Unable to select hyperslab."); + } + + // Create memory dataspace matching the count dimensions + std::vector memDims(count.begin(), count.end()); + hid_t memSpaceId = H5Screate_simple(static_cast(memDims.size()), memDims.data(), nullptr); + if(memSpaceId < 0) + { + H5Sclose(fileSpaceId); + return MakeErrorResult(-1013, "writeSpanHyperslab error: Unable to create memory dataspace."); + } + + herr_t error = H5Dwrite(datasetId, dataType, memSpaceId, fileSpaceId, H5P_DEFAULT, values.data()); + + H5Sclose(memSpaceId); + H5Sclose(fileSpaceId); + + if(error < 0) + { + return MakeErrorResult(-1014, fmt::format("writeSpanHyperslab error: H5Dwrite failed with error {}", error)); + } + + return {}; +} + template nx::core::Result DatasetIO::initChunkedDataset(const DimsType& h5Dims, const DimsType& chunkDims) const { @@ -1592,4 +1675,29 @@ template SIMPLNX_EXPORT Result<> DatasetIO::writeChunk(const ChunkedDataIn #ifdef _WIN32 template SIMPLNX_EXPORT Result<> DatasetIO::writeChunk(const ChunkedDataInfo&, const DimsType&, nonstd::span, const DimsType&, const DimsType&, nonstd::span); #endif + +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); +template SIMPLNX_EXPORT Result<> DatasetIO::createEmptyDataset(const DimsType&); + +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); +template SIMPLNX_EXPORT Result<> DatasetIO::writeSpanHyperslab(nonstd::span, const std::vector&, const std::vector&); + } // namespace nx::core::HDF5 diff --git a/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.hpp b/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.hpp index 3a01eb6717..05f0ccc883 100644 --- a/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.hpp +++ b/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.hpp @@ -205,6 +205,45 @@ class SIMPLNX_EXPORT DatasetIO : public ObjectIO template nx::core::Result<> writeSpan(const DimsType& dims, nonstd::span values); + /** + * @brief Creates an HDF5 dataset with the correct type and dimensions but does + * not write any data. This is used by out-of-core (OOC) stores that cannot + * call writeSpan() because the entire array is not resident in memory. Instead, + * OOC stores first create the empty dataset, then fill it region-by-region using + * writeSpanHyperslab() as they stream data from the backing file. + * + * In-core stores do not need this method — they use writeSpan(), which creates + * the dataset and writes all data in a single call. + * + * @tparam T The element type of the dataset + * @param dims The N-D dimensions of the dataset to create + * @return Result indicating success or failure + */ + template + nx::core::Result<> createEmptyDataset(const DimsType& dims); + + /** + * @brief Writes a contiguous span of values into a sub-region (hyperslab) of an + * existing HDF5 dataset. The dataset must already exist, created either by + * createEmptyDataset() or writeSpan(). + * + * This method exists for out-of-core (OOC) stores that materialize their data + * in chunks: the store reads a region from its backing file into a temporary + * buffer, then writes that buffer into the corresponding hyperslab of the output + * dataset. This is repeated for each region until the entire dataset is filled. + * + * In-core stores do not need this method — they use writeSpan(), which writes + * the full array in one call. + * + * @tparam T The element type of the dataset + * @param values The data to write into the hyperslab + * @param start N-D start offset for the hyperslab selection + * @param count N-D extent of the hyperslab in each dimension + * @return Result indicating success or failure + */ + template + nx::core::Result<> writeSpanHyperslab(nonstd::span values, const std::vector& start, const std::vector& count); + template Result initChunkedDataset(const DimsType& dims, const DimsType& chunkDims) const; nx::core::Result<> closeChunkedDataset(const ChunkedDataInfo& datasetInfo) const; diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index aae13f39c1..4c668b6804 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -25,17 +25,20 @@ add_executable(simplnx_test ArgumentsTest.cpp BitTest.cpp DataArrayTest.cpp + DataIOCollectionHooksTest.cpp DataPathTest.cpp DataStructObserver.hpp DataStructObserver.cpp DataStructTest.cpp DynamicFilterInstantiationTest.cpp + EmptyStringStoreTest.cpp FilePathGeneratorTest.cpp GeometryTest.cpp GeometryTestUtilities.hpp H5Test.cpp IntersectionUtilitiesTest.cpp IOFormat.cpp + IParallelAlgorithmTest.cpp MontageTest.cpp PluginTest.cpp ParametersTest.cpp diff --git a/test/DataIOCollectionHooksTest.cpp b/test/DataIOCollectionHooksTest.cpp new file mode 100644 index 0000000000..45b5fdf7b2 --- /dev/null +++ b/test/DataIOCollectionHooksTest.cpp @@ -0,0 +1,44 @@ +#include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" + +#include + +using namespace nx::core; + +TEST_CASE("DataIOCollectionHooks: format resolver default behavior") +{ + DataIOCollection collection; + REQUIRE(collection.hasFormatResolver() == false); + + // With no resolver set, should return empty string + std::string result = collection.resolveFormat(DataType::float32, {100, 100, 100}, {1}, 4000000); + REQUIRE(result.empty()); +} + +TEST_CASE("DataIOCollectionHooks: format resolver returns plugin result") +{ + DataIOCollection collection; + collection.setFormatResolver([](DataType, const ShapeType&, const ShapeType&, uint64 sizeBytes) -> std::string { return sizeBytes > 1000 ? "HDF5-OOC" : ""; }); + REQUIRE(collection.hasFormatResolver() == true); + + // Small array - returns "" + REQUIRE(collection.resolveFormat(DataType::float32, {10}, {1}, 40).empty()); + // Large array - returns "HDF5-OOC" + REQUIRE(collection.resolveFormat(DataType::float32, {1000}, {1}, 4000) == "HDF5-OOC"); +} + +TEST_CASE("DataIOCollectionHooks: format resolver can be unset") +{ + DataIOCollection collection; + collection.setFormatResolver([](DataType, const ShapeType&, const ShapeType&, uint64) -> std::string { return "HDF5-OOC"; }); + REQUIRE(collection.hasFormatResolver() == true); + + collection.setFormatResolver(nullptr); + REQUIRE(collection.hasFormatResolver() == false); + REQUIRE(collection.resolveFormat(DataType::float32, {10}, {1}, 40).empty()); +} + +TEST_CASE("DataIOCollectionHooks: backfill handler default behavior") +{ + DataIOCollection collection; + REQUIRE(collection.hasBackfillHandler() == false); +} diff --git a/test/EmptyStringStoreTest.cpp b/test/EmptyStringStoreTest.cpp new file mode 100644 index 0000000000..3b00766836 --- /dev/null +++ b/test/EmptyStringStoreTest.cpp @@ -0,0 +1,60 @@ +#include "simplnx/DataStructure/EmptyStringStore.hpp" +#include "simplnx/DataStructure/StringStore.hpp" + +#include + +using namespace nx::core; + +TEST_CASE("EmptyStringStore: basic metadata") +{ + ShapeType tupleShape = {5}; + EmptyStringStore store(tupleShape); + + REQUIRE(store.size() == 5); + REQUIRE(store.getNumberOfTuples() == 5); + REQUIRE(store.getTupleShape() == tupleShape); + REQUIRE(store.empty() == false); + REQUIRE(store.isPlaceholder() == true); +} + +TEST_CASE("EmptyStringStore: zero tuples") +{ + EmptyStringStore store({0}); + REQUIRE(store.size() == 0); + REQUIRE(store.empty() == true); + REQUIRE(store.isPlaceholder() == true); +} + +TEST_CASE("EmptyStringStore: data access throws") +{ + EmptyStringStore store({3}); + + REQUIRE_THROWS_AS(store[0], std::runtime_error); + REQUIRE_THROWS_AS(store.at(0), std::runtime_error); + REQUIRE_THROWS_AS(store.getValue(0), std::runtime_error); + REQUIRE_THROWS_AS(store.setValue(0, "test"), std::runtime_error); +} + +TEST_CASE("EmptyStringStore: deep copy preserves placeholder status") +{ + EmptyStringStore original({4}); + auto copy = original.deepCopy(); + + REQUIRE(copy->isPlaceholder() == true); + REQUIRE(copy->size() == 4); + REQUIRE(copy->getTupleShape() == ShapeType{4}); +} + +TEST_CASE("EmptyStringStore: resize") +{ + EmptyStringStore store({2}); + store.resizeTuples({10}); + REQUIRE(store.getNumberOfTuples() == 10); + REQUIRE(store.size() == 10); +} + +TEST_CASE("StringStore: isPlaceholder returns false") +{ + StringStore store(std::vector{"a", "b", "c"}, ShapeType{3}); + REQUIRE(store.isPlaceholder() == false); +} diff --git a/test/IOFormat.cpp b/test/IOFormat.cpp index 0aac8b4e56..2ba0f5c37e 100644 --- a/test/IOFormat.cpp +++ b/test/IOFormat.cpp @@ -2,6 +2,8 @@ #include "simplnx/Core/Application.hpp" #include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" +#include "simplnx/DataStructure/IO/Generic/IDataIOManager.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/MemoryUtilities.hpp" using namespace nx::core; @@ -23,18 +25,117 @@ TEST_CASE("Memory Check", "IOTest") REQUIRE(storage.free > 0); } -TEST_CASE("Target DataStructure Size", "IOTest") +// ============================================================================= +// Data Format Preference Tests (in-core only, no OOC plugin) +// ============================================================================= + +TEST_CASE("Data Format: Not configured defaults to InMemory store", "[IOTest][DataFormat]") +{ + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); + + // With no OOC plugin loaded and no format configured, largeDataFormat() + // should return empty (not configured) and useOocData() should be false. + std::string savedFormat = prefs->largeDataFormat(); + prefs->setLargeDataFormat(""); + REQUIRE(prefs->largeDataFormat().empty()); + REQUIRE_FALSE(prefs->useOocData()); + + // CreateDataStore should produce an InMemory store regardless of size + auto store = DataStoreUtilities::CreateDataStore({100, 100, 100}, {1}, IDataAction::Mode::Execute); + REQUIRE(store != nullptr); + REQUIRE(store->getStoreType() == IDataStore::StoreType::InMemory); + + prefs->setLargeDataFormat(savedFormat); +} + +TEST_CASE("Data Format: Explicit InMemory format prevents OOC", "[IOTest][DataFormat]") +{ + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); + + std::string savedFormat = prefs->largeDataFormat(); + prefs->setLargeDataFormat(std::string(Preferences::k_InMemoryFormat)); + + // k_InMemoryFormat is non-empty but should NOT enable OOC + REQUIRE_FALSE(prefs->largeDataFormat().empty()); + REQUIRE(prefs->largeDataFormat() == Preferences::k_InMemoryFormat); + REQUIRE_FALSE(prefs->useOocData()); + + // CreateDataStore should produce InMemory even for large arrays + auto store = DataStoreUtilities::CreateDataStore({100, 100, 100}, {1}, IDataAction::Mode::Execute); + REQUIRE(store != nullptr); + REQUIRE(store->getStoreType() == IDataStore::StoreType::InMemory); + + prefs->setLargeDataFormat(savedFormat); +} + +TEST_CASE("Data Format: checkUseOoc returns false for empty string", "[IOTest][DataFormat]") +{ + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); + + std::string savedFormat = prefs->largeDataFormat(); + prefs->setLargeDataFormat(""); + REQUIRE_FALSE(prefs->useOocData()); + prefs->setLargeDataFormat(savedFormat); +} + +TEST_CASE("Data Format: checkUseOoc returns false for InMemory format", "[IOTest][DataFormat]") { - auto* preferences = Application::GetOrCreateInstance()->getPreferences(); - REQUIRE(preferences->largeDataStructureSize() > 0); + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); - const uint64 memory = Memory::GetTotalMemory(); - const uint64 largeDataSize = preferences->valueAs(Preferences::k_LargeDataSize_Key); - const uint64 minimumRemaining = 2 * largeDataSize; - uint64 targetReducedSize = (memory - 2 * largeDataSize); - if(minimumRemaining >= memory) + std::string savedFormat = prefs->largeDataFormat(); + prefs->setLargeDataFormat(std::string(Preferences::k_InMemoryFormat)); + REQUIRE_FALSE(prefs->useOocData()); + prefs->setLargeDataFormat(savedFormat); +} + +TEST_CASE("Data Format: Cannot register IO manager with reserved InMemory name", "[IOTest][DataFormat]") +{ + // Create a dummy IDataIOManager subclass that returns k_InMemoryFormat + class ReservedNameManager : public IDataIOManager { - targetReducedSize = memory / 2; - } - REQUIRE(preferences->defaultValueAs(Preferences::k_LargeDataStructureSize_Key) == targetReducedSize); + public: + std::string formatName() const override + { + return std::string(Preferences::k_InMemoryFormat); + } + }; + + auto ioCollection = Application::GetOrCreateInstance()->getIOCollection(); + auto badManager = std::make_shared(); + REQUIRE_THROWS(ioCollection->addIOManager(badManager)); +} + +TEST_CASE("Data Format: resolveFormat returns empty with no resolver registered", "[IOTest][DataFormat]") +{ + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); + auto ioCollection = Application::GetOrCreateInstance()->getIOCollection(); + + std::string savedFormat = prefs->largeDataFormat(); + + // Set format to InMemory — core must not redirect to any OOC format + prefs->setLargeDataFormat(std::string(Preferences::k_InMemoryFormat)); + + // With no OOC plugin loaded, no resolver is registered and resolveFormat returns "" + uint64 largeSize = 2ULL * 1024 * 1024 * 1024; // 2 GB — well above any threshold + std::string dataFormat = ioCollection->resolveFormat(DataType::float32, {100, 100, 100}, {1}, largeSize); + REQUIRE(dataFormat.empty()); // Should NOT be changed to any OOC format + + prefs->setLargeDataFormat(savedFormat); +} + +TEST_CASE("Data Format: resolveFormat returns empty when format not configured", "[IOTest][DataFormat]") +{ + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); + auto ioCollection = Application::GetOrCreateInstance()->getIOCollection(); + + std::string savedFormat = prefs->largeDataFormat(); + + // Empty format = not configured, and no OOC plugin is loaded in the in-core build + prefs->setLargeDataFormat(""); + + uint64 largeSize = 2ULL * 1024 * 1024 * 1024; + std::string dataFormat = ioCollection->resolveFormat(DataType::float32, {100, 100, 100}, {1}, largeSize); + REQUIRE(dataFormat.empty()); // No OOC format available + + prefs->setLargeDataFormat(savedFormat); } diff --git a/test/IParallelAlgorithmTest.cpp b/test/IParallelAlgorithmTest.cpp new file mode 100644 index 0000000000..426b213ec5 --- /dev/null +++ b/test/IParallelAlgorithmTest.cpp @@ -0,0 +1,296 @@ +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataStore.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Utilities/ParallelDataAlgorithm.hpp" + +#include + +#include +#include + +using namespace nx::core; + +namespace +{ +/** + * @brief Minimal mock data store that reports StoreType::OutOfCore. + * + * Only getStoreType() is meaningful; every other method throws because + * the tests never access actual element data. + */ +template +class MockOocDataStore : public AbstractDataStore +{ +public: + using value_type = typename AbstractDataStore::value_type; + + MockOocDataStore(const ShapeType& tupleShape, const ShapeType& componentShape) + : m_TupleShape(tupleShape) + , m_ComponentShape(componentShape) + , m_NumTuples(std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>())) + , m_NumComponents(std::accumulate(componentShape.cbegin(), componentShape.cend(), static_cast(1), std::multiplies<>())) + { + } + + ~MockOocDataStore() override = default; + + IDataStore::StoreType getStoreType() const override + { + return IDataStore::StoreType::OutOfCore; + } + + usize getNumberOfTuples() const override + { + return m_NumTuples; + } + + usize getNumberOfComponents() const override + { + return m_NumComponents; + } + + const ShapeType& getTupleShape() const override + { + return m_TupleShape; + } + + const ShapeType& getComponentShape() const override + { + return m_ComponentShape; + } + + DataType getDataType() const override + { + return GetDataType(); + } + + void resizeTuples(const ShapeType& /*tupleShape*/) override + { + throw std::runtime_error("MockOocDataStore::resizeTuples not implemented"); + } + + value_type getValue(usize /*index*/) const override + { + throw std::runtime_error("MockOocDataStore::getValue not implemented"); + } + + void setValue(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::setValue not implemented"); + } + + void copyIntoBuffer(usize /*startIndex*/, nonstd::span /*buffer*/) const override + { + throw std::runtime_error("MockOocDataStore::copyIntoBuffer not implemented"); + } + + void copyFromBuffer(usize /*startIndex*/, nonstd::span /*buffer*/) override + { + throw std::runtime_error("MockOocDataStore::copyFromBuffer not implemented"); + } + + value_type at(usize /*index*/) const override + { + throw std::runtime_error("MockOocDataStore::at not implemented"); + } + + void add(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::add not implemented"); + } + + void sub(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::sub not implemented"); + } + + void mul(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::mul not implemented"); + } + + void div(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::div not implemented"); + } + + void rem(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::rem not implemented"); + } + + void bitwiseAND(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::bitwiseAND not implemented"); + } + + void bitwiseOR(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::bitwiseOR not implemented"); + } + + void bitwiseXOR(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::bitwiseXOR not implemented"); + } + + void bitwiseLShift(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::bitwiseLShift not implemented"); + } + + void bitwiseRShift(usize /*index*/, value_type /*value*/) override + { + throw std::runtime_error("MockOocDataStore::bitwiseRShift not implemented"); + } + + void byteSwap(usize /*index*/) override + { + throw std::runtime_error("MockOocDataStore::byteSwap not implemented"); + } + + void swap(usize /*index1*/, usize /*index2*/) override + { + throw std::runtime_error("MockOocDataStore::swap not implemented"); + } + + std::unique_ptr deepCopy() const override + { + return std::make_unique(*this); + } + + std::unique_ptr createNewInstance() const override + { + return std::make_unique(m_TupleShape, m_ComponentShape); + } + + std::pair writeBinaryFile(const std::string& /*absoluteFilePath*/) const override + { + return {-1, "MockOocDataStore cannot write files"}; + } + + std::pair writeBinaryFile(std::ostream& /*outputStream*/) const override + { + return {-1, "MockOocDataStore cannot write files"}; + } + + Result<> readHdf5(const HDF5::DatasetIO& /*dataset*/) override + { + return MakeErrorResult(-1, "MockOocDataStore cannot read HDF5"); + } + + Result<> writeHdf5(HDF5::DatasetIO& /*dataset*/) const override + { + return MakeErrorResult(-1, "MockOocDataStore cannot write HDF5"); + } + +private: + ShapeType m_TupleShape; + ShapeType m_ComponentShape; + usize m_NumTuples = 0; + usize m_NumComponents = 0; +}; +} // namespace + +TEST_CASE("IParallelAlgorithm: TBB enabled by default", "[simplnx][IParallelAlgorithm]") +{ + ParallelDataAlgorithm algorithm; + +#ifdef SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == true); +#else + REQUIRE(algorithm.getParallelizationEnabled() == false); +#endif +} + +TEST_CASE("IParallelAlgorithm: requireArraysInMemory with in-memory arrays keeps TBB enabled", "[simplnx][IParallelAlgorithm]") +{ + DataStructure dataStructure; + auto store = std::make_shared>(ShapeType{10}, ShapeType{1}, 0.0f); + auto* dataArray = DataArray::Create(dataStructure, "TestArray", store); + REQUIRE(dataArray != nullptr); + + ParallelDataAlgorithm algorithm; + algorithm.requireArraysInMemory({dataArray}); + +#ifdef SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == true); +#else + REQUIRE(algorithm.getParallelizationEnabled() == false); +#endif +} + +TEST_CASE("IParallelAlgorithm: requireStoresInMemory with in-memory stores keeps TBB enabled", "[simplnx][IParallelAlgorithm]") +{ + DataStore store(ShapeType{10}, ShapeType{1}, 0.0f); + + ParallelDataAlgorithm algorithm; + algorithm.requireStoresInMemory({&store}); + +#ifdef SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == true); +#else + REQUIRE(algorithm.getParallelizationEnabled() == false); +#endif +} + +TEST_CASE("IParallelAlgorithm: requireArraysInMemory with OOC arrays disables TBB", "[simplnx][IParallelAlgorithm]") +{ + DataStructure dataStructure; + auto oocStore = std::make_shared>(ShapeType{10}, ShapeType{1}); + auto* dataArray = DataArray::Create(dataStructure, "OocArray", oocStore); + REQUIRE(dataArray != nullptr); + + ParallelDataAlgorithm algorithm; + algorithm.requireArraysInMemory({dataArray}); + + // OOC arrays should disable parallelization regardless of SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == false); +} + +TEST_CASE("IParallelAlgorithm: requireStoresInMemory with OOC stores disables TBB", "[simplnx][IParallelAlgorithm]") +{ + MockOocDataStore oocStore(ShapeType{10}, ShapeType{1}); + + ParallelDataAlgorithm algorithm; + algorithm.requireStoresInMemory({&oocStore}); + + // OOC stores should disable parallelization regardless of SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == false); +} + +TEST_CASE("IParallelAlgorithm: requireStoresInMemory with mixed stores disables TBB", "[simplnx][IParallelAlgorithm]") +{ + DataStore inMemoryStore(ShapeType{10}, ShapeType{1}, 0.0f); + MockOocDataStore oocStore(ShapeType{10}, ShapeType{1}); + + ParallelDataAlgorithm algorithm; + algorithm.requireStoresInMemory({&inMemoryStore, &oocStore}); + + // A single OOC store in the mix should disable parallelization + REQUIRE(algorithm.getParallelizationEnabled() == false); +} + +TEST_CASE("IParallelAlgorithm: requireArraysInMemory with empty array list keeps TBB enabled", "[simplnx][IParallelAlgorithm]") +{ + ParallelDataAlgorithm algorithm; + algorithm.requireArraysInMemory({}); + +#ifdef SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == true); +#else + REQUIRE(algorithm.getParallelizationEnabled() == false); +#endif +} + +TEST_CASE("IParallelAlgorithm: requireStoresInMemory with nullptr entries keeps TBB enabled", "[simplnx][IParallelAlgorithm]") +{ + ParallelDataAlgorithm algorithm; + algorithm.requireStoresInMemory({nullptr, nullptr}); + +#ifdef SIMPLNX_ENABLE_MULTICORE + REQUIRE(algorithm.getParallelizationEnabled() == true); +#else + REQUIRE(algorithm.getParallelizationEnabled() == false); +#endif +} diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp index 67572b5159..891e39c8d8 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp @@ -25,6 +25,7 @@ #include "simplnx/Parameters/DataTypeParameter.hpp" #include "simplnx/Parameters/GeometrySelectionParameter.hpp" #include "simplnx/Utilities/DataGroupUtilities.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/MD5.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" @@ -456,45 +457,73 @@ inline void CompareMontage(const AbstractMontage& exemplar, const AbstractMontag } /** - * @brief Compares IDataArray - * @tparam T - * @param left - * @param right + * @brief Compares two IDataArrays using bulk copyIntoBuffer for OOC efficiency. + * + * Reads both arrays in 40K-element chunks via copyIntoBuffer instead of + * per-element operator[], avoiding per-voxel OOC store overhead. + * NaN == NaN is treated as equal for floating-point types. + * + * @tparam T The element type + * @param left First array (typically exemplar) + * @param right Second array (typically generated) + * @param start Element index to start comparison from (default 0) */ template void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize start = 0) { const auto& oldDataStore = left.template getIDataStoreRefAs>(); const auto& newDataStore = right.template getIDataStoreRefAs>(); - usize end = oldDataStore.getSize(); + const usize totalSize = oldDataStore.getSize(); INFO(fmt::format("Input Data Array:'{}' Output DataArray: '{}' bad comparison", left.getName(), right.getName())); - T oldVal; - T newVal; + REQUIRE(totalSize == newDataStore.getSize()); + + constexpr usize k_ChunkSize = 40000; + auto oldBuf = std::make_unique(k_ChunkSize); + auto newBuf = std::make_unique(k_ChunkSize); + bool failed = false; - for(usize i = start; i < end; i++) + usize failIndex = 0; + T failOld = {}; + T failNew = {}; + + for(usize offset = start; offset < totalSize && !failed; offset += k_ChunkSize) { - oldVal = oldDataStore[i]; - newVal = newDataStore[i]; - if(oldVal != newVal) - { - UNSCOPED_INFO(fmt::format("index=: {} oldValue != newValue. {} != {}", i, oldVal, newVal)); + const usize count = std::min(k_ChunkSize, totalSize - offset); + oldDataStore.copyIntoBuffer(offset, nonstd::span(oldBuf.get(), count)); + newDataStore.copyIntoBuffer(offset, nonstd::span(newBuf.get(), count)); - if constexpr(std::is_floating_point_v) + for(usize i = 0; i < count; i++) + { + const T oldVal = oldBuf[i]; + const T newVal = newBuf[i]; + if(oldVal != newVal) { - float diff = std::fabs(static_cast(oldVal - newVal)); - if(diff > EPSILON) + if constexpr(std::is_floating_point_v) { - failed = true; - break; + // NaN == NaN is treated as equal + if(std::isnan(oldVal) && std::isnan(newVal)) + { + continue; + } + float32 diff = std::fabs(static_cast(oldVal - newVal)); + if(diff <= EPSILON) + { + continue; + } } - } - else - { failed = true; + failIndex = offset + i; + failOld = oldVal; + failNew = newVal; + break; } - break; } } + + if(failed) + { + UNSCOPED_INFO(fmt::format("index=: {} oldValue != newValue. {} != {}", failIndex, failOld, failNew)); + } REQUIRE(!failed); } From 8eb2ad4c2a608c772436d1b6522bd9ac8dd6d8d6 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Wed, 8 Apr 2026 13:41:02 -0400 Subject: [PATCH 02/13] DOC: Add thorough Doxygen and inline documentation for OOC architecture Add comprehensive documentation to all new methods, type aliases, classes, and algorithms introduced in the OOC architecture rewrite. Every new public API now has Doxygen explaining what it does, how it works, and why it is needed. Algorithm implementations have step-by- step inline comments explaining the logic. Signed-off-by: Joey Kleingers --- .../src/SimplnxCore/utils/VtkUtilities.hpp | 29 +- src/simplnx/Core/Preferences.cpp | 40 ++- src/simplnx/Core/Preferences.hpp | 113 ++++-- .../DataStructure/AbstractDataStore.hpp | 53 ++- src/simplnx/DataStructure/DataStore.hpp | 28 ++ src/simplnx/DataStructure/EmptyDataStore.hpp | 36 +- src/simplnx/DataStructure/IDataStore.hpp | 56 ++- .../IO/Generic/DataIOCollection.cpp | 68 ++++ .../IO/Generic/DataIOCollection.hpp | 339 +++++++++++++----- .../IO/Generic/IDataIOManager.cpp | 27 ++ .../IO/Generic/IDataIOManager.hpp | 214 ++++++++--- .../DataStructure/IO/HDF5/DataStoreIO.hpp | 131 +++++-- .../IO/HDF5/DataStructureWriter.cpp | 17 +- .../IO/HDF5/DataStructureWriter.hpp | 18 +- .../DataStructure/IO/HDF5/NeighborListIO.hpp | 78 ++-- .../DataStructure/IO/HDF5/StringArrayIO.cpp | 10 +- src/simplnx/Utilities/AlgorithmDispatch.hpp | 57 ++- src/simplnx/Utilities/DataStoreUtilities.hpp | 56 ++- .../Utilities/Parsing/DREAM3D/Dream3dIO.cpp | 32 ++ .../Utilities/Parsing/DREAM3D/Dream3dIO.hpp | 31 +- .../Utilities/Parsing/HDF5/IO/DatasetIO.cpp | 48 ++- .../simplnx/UnitTest/UnitTestCommon.hpp | 46 ++- 22 files changed, 1235 insertions(+), 292 deletions(-) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp index 5cae085711..5350e4efab 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp @@ -134,6 +134,9 @@ std::string TypeForPrimitive(const IFilter::MessageHandler& messageHandler) return ""; } +// ----------------------------------------------------------------------------- +// Functor for writing a DataArray to a VTK legacy file via FILE* I/O. +// Supports both binary (big-endian) and ASCII output modes. // ----------------------------------------------------------------------------- struct WriteVtkDataArrayFunctor { @@ -161,15 +164,31 @@ struct WriteVtkDataArrayFunctor fprintf(outputFile, "LOOKUP_TABLE default\n"); if(binary) { - // Read data into buffer, byte-swap in memory, and write. - // This avoids modifying the DataStore (critical for OOC stores - // where per-element byte-swap via setValue would be very slow). + // --------------------------------------------------------------- + // Chunked binary write pattern for OOC compatibility + // --------------------------------------------------------------- + // The original code used dataArray->data() for a single fwrite, + // which requires the entire array to be resident in memory. This + // fails for OOC stores where data lives on disk and data() is + // not available. + // + // Instead, we read 4096 elements at a time into a local buffer + // via copyIntoBuffer (the OOC-compatible bulk read API), perform + // an in-place byte swap to big-endian (VTK legacy binary format + // requires big-endian), and fwrite the buffer. This keeps memory + // usage constant regardless of array size. + // + // For bool arrays, copyIntoBuffer is not available (bool is not + // a supported span type), so we use per-element getValue() and + // convert to uint8 (0 or 1). + // --------------------------------------------------------------- constexpr usize k_ChunkSize = 4096; for(usize offset = 0; offset < totalElements; offset += k_ChunkSize) { usize count = std::min(k_ChunkSize, totalElements - offset); if constexpr(std::is_same_v) { + // Bool special case: convert to uint8 via per-element access. std::vector buf(count); for(usize i = 0; i < count; i++) { @@ -179,10 +198,14 @@ struct WriteVtkDataArrayFunctor } else { + // General case: bulk read into buffer, byte-swap, then write. std::vector buf(count); dataStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); if constexpr(endian::little == endian::native) { + // VTK legacy binary requires big-endian. Swap in the local + // buffer rather than mutating the DataStore, which would be + // slow for OOC stores and would modify shared data. for(usize i = 0; i < count; i++) { buf[i] = nx::core::byteswap(buf[i]); diff --git a/src/simplnx/Core/Preferences.cpp b/src/simplnx/Core/Preferences.cpp index 404ea32451..bc5ff09511 100644 --- a/src/simplnx/Core/Preferences.cpp +++ b/src/simplnx/Core/Preferences.cpp @@ -120,14 +120,17 @@ void Preferences::setLargeDataFormat(std::string dataFormat) if(dataFormat.empty()) { // Remove the key so the default (set by plugins) can take effect. - // An empty string means "not configured", not "in-core" — use - // k_InMemoryFormat for an explicit in-core choice. + // An empty string means "not configured", not "in-core". To explicitly + // request in-core storage, pass k_InMemoryFormat instead. This distinction + // matters because the SimplnxOoc plugin sets a default OOC format on startup, + // and erasing the user value lets that default take effect. m_Values.erase(k_PreferredLargeDataFormat_Key); } else { m_Values[k_PreferredLargeDataFormat_Key] = dataFormat; } + // Recompute the cached m_UseOoc flag after any format change checkUseOoc(); } @@ -271,9 +274,14 @@ Result<> Preferences::loadFromFile(const std::filesystem::path& filepath) m_Values = parsedResult; - // Migrate legacy format strings from saved preferences: - // - Empty string: legacy "not configured" state → remove so plugin defaults take effect - // - "In-Memory": legacy explicit in-core choice → remove so plugin defaults take effect + // Migrate legacy format strings from saved preferences files that were + // written before the OOC architecture was finalized. Two legacy values + // need cleanup: + // - Empty string (""): Old "not configured" state. Removing the key + // lets the plugin-supplied default (e.g., "HDF5-OOC") take effect. + // - "In-Memory": Old explicit in-core sentinel. Replaced by + // k_InMemoryFormat ("Simplnx-Default-In-Memory"). Removing the key + // avoids confusion with the new sentinel value. if(m_Values.contains(k_PreferredLargeDataFormat_Key) && m_Values[k_PreferredLargeDataFormat_Key].is_string()) { const std::string savedFormat = m_Values[k_PreferredLargeDataFormat_Key].get(); @@ -283,6 +291,7 @@ Result<> Preferences::loadFromFile(const std::filesystem::path& filepath) } } + // Recompute derived state from the loaded (and possibly migrated) values checkUseOoc(); updateMemoryDefaults(); return {}; @@ -290,12 +299,19 @@ Result<> Preferences::loadFromFile(const std::filesystem::path& filepath) void Preferences::checkUseOoc() { + // Resolve the format from user values first, then default values (via value()) auto formatJson = value(k_PreferredLargeDataFormat_Key); + + // If no format is configured (null/non-string), OOC is not active if(formatJson.is_null() || !formatJson.is_string()) { m_UseOoc = false; return; } + + // OOC is active when the format is a non-empty string that is NOT the + // explicit in-memory sentinel. This means a plugin (e.g., SimplnxOoc) + // has registered a real OOC format like "HDF5-OOC". const std::string format = formatJson.get(); m_UseOoc = !format.empty() && format != k_InMemoryFormat; } @@ -325,9 +341,15 @@ void Preferences::setForceOocData(bool forceOoc) void Preferences::updateMemoryDefaults() { + // Reserve headroom equal to 2x the single-array large-data threshold. + // This leaves room for the OS, the application, and at least one large + // array being constructed while the DataStructure holds existing data. const uint64 minimumRemaining = 2 * defaultValueAs(k_LargeDataSize_Key); const uint64 totalMemory = Memory::GetTotalMemory(); uint64 targetValue = totalMemory - minimumRemaining; + + // On low-memory systems where the reservation exceeds total RAM, + // fall back to using half of total RAM as the threshold if(minimumRemaining >= totalMemory) { targetValue = totalMemory / 2; @@ -368,8 +390,12 @@ void Preferences::setOocRangeScanTimeoutSeconds(uint32 seconds) uint64 Preferences::oocMemoryBudgetBytes() const { - // Default: 8 GB (the application will set the real default from - // OocMemoryBudgetManager::defaultBudgetBytes() on startup) + // Hard-coded fallback of 8 GB. This conservative default is used when: + // 1. The user has never saved an explicit budget preference, AND + // 2. The SimplnxOoc plugin has not yet called setOocMemoryBudgetBytes() + // with its computed default (50% of system RAM). + // Using m_Values.value() (not the value() member) reads directly from + // user-set values with the fallback, bypassing the default-value layer. static constexpr uint64 k_DefaultBudget = 8ULL * 1024 * 1024 * 1024; return m_Values.value(k_OocMemoryBudgetBytes_Key, k_DefaultBudget); } diff --git a/src/simplnx/Core/Preferences.hpp b/src/simplnx/Core/Preferences.hpp index 5c02226266..a1022ff446 100644 --- a/src/simplnx/Core/Preferences.hpp +++ b/src/simplnx/Core/Preferences.hpp @@ -24,14 +24,49 @@ class SIMPLNX_EXPORT Preferences friend class AbstractPlugin; public: - static inline constexpr StringLiteral k_LargeDataSize_Key = "large_data_size"; // bytes - static inline constexpr StringLiteral k_PreferredLargeDataFormat_Key = "large_data_format"; // string - static inline constexpr StringLiteral k_InMemoryFormat = "Simplnx-Default-In-Memory"; // explicit in-core; empty string means "not configured" - static inline constexpr StringLiteral k_LargeDataStructureSize_Key = "large_datastructure_size"; // bytes - static inline constexpr StringLiteral k_ForceOocData_Key = "force_ooc_data"; // boolean - static inline constexpr nx::core::StringLiteral k_OoCTempDirectory_ID = "ooc_temp_directory"; // Out-of-Core temp directory - static inline constexpr StringLiteral k_OocRangeScanTimeoutSeconds_Key = "ooc_range_scan_timeout_seconds"; // uint32, seconds - static inline constexpr StringLiteral k_OocMemoryBudgetBytes_Key = "ooc_memory_budget_bytes"; // uint64, bytes + /// @name Preference Keys + /// JSON keys used to store and retrieve preference values. These keys appear + /// in the serialized preferences.json file and are used internally by the + /// getter/setter methods below. + /// @{ + + /// Byte-size threshold above which a single DataArray is considered "large" + /// and may be written to an OOC-capable format instead of in-memory storage. + static inline constexpr StringLiteral k_LargeDataSize_Key = "large_data_size"; + + /// Name of the preferred storage format for large DataArrays (e.g., "HDF5-OOC"). + /// An empty string means "not yet configured by the user or plugin". + static inline constexpr StringLiteral k_PreferredLargeDataFormat_Key = "large_data_format"; + + /// Sentinel value for k_PreferredLargeDataFormat_Key that explicitly requests + /// in-memory storage. This is distinct from an empty string, which means + /// "not configured" and falls back to plugin-supplied defaults. + static inline constexpr StringLiteral k_InMemoryFormat = "Simplnx-Default-In-Memory"; + + /// Byte-size threshold for the entire DataStructure. When total memory usage + /// approaches this value, the application may switch to OOC storage for new arrays. + /// The default is computed dynamically by updateMemoryDefaults() based on system RAM. + static inline constexpr StringLiteral k_LargeDataStructureSize_Key = "large_datastructure_size"; + + /// Boolean flag that, when true, forces all new DataArrays to use OOC storage + /// regardless of their size. Only takes effect when an OOC format is active. + static inline constexpr StringLiteral k_ForceOocData_Key = "force_ooc_data"; + + /// Filesystem path to the directory where OOC temporary files (chunk stores, + /// backing HDF5 files) are created during filter execution. + static inline constexpr nx::core::StringLiteral k_OoCTempDirectory_ID = "ooc_temp_directory"; + + /// Timeout in seconds for the background thread that scans OOC DataArrays to + /// compute their value ranges (min/max). If the scan does not complete within + /// this window, the range is reported as unknown. Default: 30 seconds. + static inline constexpr StringLiteral k_OocRangeScanTimeoutSeconds_Key = "ooc_range_scan_timeout_seconds"; + + /// Total memory budget in bytes shared across all OOC caching subsystems + /// (chunk cache, stride cache, partition cache). The OOC memory budget manager + /// distributes this budget via global LRU eviction. Default: 8 GB. + static inline constexpr StringLiteral k_OocMemoryBudgetBytes_Key = "ooc_memory_budget_bytes"; + + /// @} /** * @brief Returns the default file path for storing preferences based on the application name. @@ -230,7 +265,15 @@ class SIMPLNX_EXPORT Preferences void setForceOocData(bool forceOoc); /** - * @brief Updates memory-related default values based on system capabilities. + * @brief Recomputes the default value for k_LargeDataStructureSize_Key based + * on the current system's total physical RAM. + * + * The target value is (totalRAM - 2 * k_LargeDataSize), which reserves + * headroom for the OS and the application itself. If the reservation would + * exceed total RAM (e.g., on a low-memory system), the fallback is totalRAM / 2. + * + * Called automatically during construction, after loadFromFile(), and after + * clear(). Can also be called explicitly after changing k_LargeDataSize_Key. */ void updateMemoryDefaults(); @@ -253,32 +296,54 @@ class SIMPLNX_EXPORT Preferences void setOocTempDirectory(const std::string& path); /** - * @brief Gets the timeout (in seconds) for the background OOC range scan. + * @brief Gets the timeout for the background OOC range scan. + * + * The range scan runs on a background thread after an OOC DataArray is loaded, + * computing min/max values by reading through all chunks sequentially. If the + * scan does not complete within this timeout, the range is reported as unknown + * and the UI shows "N/A" for the array's value range. + * * @return Timeout in seconds (default 30) */ uint32 oocRangeScanTimeoutSeconds() const; /** - * @brief Sets the timeout (in seconds) for the background OOC range scan. - * @param seconds Timeout value in seconds + * @brief Sets the timeout for the background OOC range scan. + * @param seconds Timeout value in seconds. A value of 0 effectively disables + * the range scan by expiring it immediately. */ void setOocRangeScanTimeoutSeconds(uint32 seconds); /** - * @brief Gets the total memory budget for all OOC caching (chunk cache, stride cache, partition cache). + * @brief Gets the total memory budget for all OOC caching subsystems. + * + * The OOC memory budget manager distributes this budget across the chunk + * cache, stride cache, and partition cache using global LRU eviction. When + * the combined memory usage of all caches exceeds this budget, the least + * recently used entries are evicted to make room for new data. * - * The budget manager distributes this across subsystems via global LRU eviction. - * The default (8 GB) is a safe fallback; the SimplnxOoc plugin overrides this on - * startup with OocMemoryBudgetManager::defaultBudgetBytes() (50% of system RAM) - * if the user has not saved an explicit preference. + * The default value (8 GB) is a conservative fallback used when no plugin + * has configured a more appropriate value. On startup, the SimplnxOoc plugin + * calls OocMemoryBudgetManager::defaultBudgetBytes() (50% of system RAM) and + * sets that as the budget, unless the user has already saved an explicit + * preference via the UI. * - * @return Budget in bytes (default 8 GB) + * @note This reads from m_Values (user-set) directly, NOT from m_DefaultValues, + * because the default is hard-coded as a compile-time constant. + * + * @return Budget in bytes (default 8 GB if not explicitly set) */ uint64 oocMemoryBudgetBytes() const; /** - * @brief Sets the total memory budget for all OOC caching. - * @param bytes Budget in bytes + * @brief Sets the total memory budget for all OOC caching subsystems. + * + * The new budget takes effect immediately for subsequent cache eviction + * decisions. Existing cached data that exceeds the new budget will be + * evicted lazily as new cache entries are requested. + * + * @param bytes Budget in bytes. Must be > 0; passing 0 would effectively + * disable caching. */ void setOocMemoryBudgetBytes(uint64 bytes); @@ -297,7 +362,13 @@ class SIMPLNX_EXPORT Preferences void addDefaultValues(std::string pluginName, std::string valueName, const nlohmann::json& value); /** - * @brief Checks and updates whether out-of-core mode should be used based on current settings. + * @brief Recomputes the cached m_UseOoc flag based on the current value of + * k_PreferredLargeDataFormat_Key. + * + * OOC mode is considered active when the resolved format string is non-empty + * and is not the sentinel value k_InMemoryFormat. This method is called after + * any operation that could change the format: construction, loadFromFile(), + * setLargeDataFormat(), and setDefaultLargeDataFormat(). */ void checkUseOoc(); diff --git a/src/simplnx/DataStructure/AbstractDataStore.hpp b/src/simplnx/DataStructure/AbstractDataStore.hpp index 43b7d1e574..2a4fe11ce0 100644 --- a/src/simplnx/DataStructure/AbstractDataStore.hpp +++ b/src/simplnx/DataStructure/AbstractDataStore.hpp @@ -412,22 +412,59 @@ class AbstractDataStore : public IDataStore /** * @brief Copies a contiguous range of values from this data store into the - * provided buffer. The buffer must be large enough to hold the requested - * range. Each store subclass implements its own optimal version: in-memory - * stores use std::copy; out-of-core stores use bulk chunk I/O. + * provided caller-owned buffer. + * + * This is the primary bulk-read API for algorithms that need to process data + * in contiguous blocks. It replaces the earlier chunk-based API and provides + * a single uniform interface that works identically for both in-memory and + * out-of-core (OOC) data stores: + * + * - **In-memory (DataStore):** Performs a direct std::copy from the backing + * array into the buffer. This is essentially zero-overhead. + * - **Out-of-core (OOC stores):** The OOC subclass translates the flat + * element range into the appropriate chunk reads from the backing HDF5 + * file, coalescing I/O where possible. The caller does not need to know + * the chunk layout. + * - **Empty (EmptyDataStore):** Throws std::runtime_error because no data + * exists. + * + * The number of elements to copy is determined by `buffer.size()`. The caller + * is responsible for ensuring the buffer is large enough and that the range + * `[startIndex, startIndex + buffer.size())` does not exceed `getSize()`. * * @param startIndex The starting flat element index to read from - * @param buffer A span to receive the copied values (buffer.size() determines count) + * @param buffer A span to receive the copied values; its size determines how + * many elements are read + * @throw std::out_of_range If the requested range exceeds the store's size + * @throw std::runtime_error If called on an EmptyDataStore */ virtual void copyIntoBuffer(usize startIndex, nonstd::span buffer) const = 0; /** - * @brief Copies values from the provided buffer into a contiguous range of - * this data store. Each store subclass implements its own optimal version: - * in-memory stores use std::copy; out-of-core stores use bulk chunk I/O. + * @brief Copies values from the provided caller-owned buffer into a + * contiguous range of this data store. + * + * This is the primary bulk-write API, the write-side counterpart of + * copyIntoBuffer(). It provides a single uniform interface for both + * in-memory and out-of-core (OOC) data stores: + * + * - **In-memory (DataStore):** Performs a direct std::copy from the buffer + * into the backing array. + * - **Out-of-core (OOC stores):** The OOC subclass translates the flat + * element range into the appropriate chunk writes to the backing HDF5 + * file. + * - **Empty (EmptyDataStore):** Throws std::runtime_error because no data + * exists. + * + * The number of elements to copy is determined by `buffer.size()`. The caller + * is responsible for ensuring the range `[startIndex, startIndex + buffer.size())` + * does not exceed `getSize()`. * * @param startIndex The starting flat element index to write to - * @param buffer A span containing the values to copy into the store + * @param buffer A span containing the values to copy into the store; its + * size determines how many elements are written + * @throw std::out_of_range If the requested range exceeds the store's size + * @throw std::runtime_error If called on an EmptyDataStore */ virtual void copyFromBuffer(usize startIndex, nonstd::span buffer) = 0; diff --git a/src/simplnx/DataStructure/DataStore.hpp b/src/simplnx/DataStructure/DataStore.hpp index cb75fa26a4..bcccd1847f 100644 --- a/src/simplnx/DataStructure/DataStore.hpp +++ b/src/simplnx/DataStructure/DataStore.hpp @@ -299,23 +299,51 @@ class DataStore : public AbstractDataStore m_Data.get()[index] = value; } + /** + * @brief Copies a contiguous range of values from this in-memory data store + * into the caller-provided buffer. For the in-memory DataStore this is a + * simple bounds-checked std::copy from the raw backing array. + * + * @param startIndex The starting flat element index to read from + * @param buffer A span to receive the copied values; its size determines how + * many elements are read + * @throw std::out_of_range If `[startIndex, startIndex + buffer.size())` exceeds getSize() + */ void copyIntoBuffer(usize startIndex, nonstd::span buffer) const override { const usize count = buffer.size(); + + // Bounds check: ensure the requested range fits within the store if(startIndex + count > this->getSize()) { throw std::out_of_range(fmt::format("DataStore::copyIntoBuffer: range [{}, {}) exceeds size {}", startIndex, startIndex + count, this->getSize())); } + + // Direct memory copy from the contiguous backing array into the caller's buffer std::copy(m_Data.get() + startIndex, m_Data.get() + startIndex + count, buffer.data()); } + /** + * @brief Copies values from the caller-provided buffer into a contiguous + * range of this in-memory data store. For the in-memory DataStore this is + * a simple bounds-checked std::copy into the raw backing array. + * + * @param startIndex The starting flat element index to write to + * @param buffer A span containing the values to write; its size determines + * how many elements are written + * @throw std::out_of_range If `[startIndex, startIndex + buffer.size())` exceeds getSize() + */ void copyFromBuffer(usize startIndex, nonstd::span buffer) override { const usize count = buffer.size(); + + // Bounds check: ensure the requested range fits within the store if(startIndex + count > this->getSize()) { throw std::out_of_range(fmt::format("DataStore::copyFromBuffer: range [{}, {}) exceeds size {}", startIndex, startIndex + count, this->getSize())); } + + // Direct memory copy from the caller's buffer into the contiguous backing array std::copy(buffer.begin(), buffer.end(), m_Data.get() + startIndex); } diff --git a/src/simplnx/DataStructure/EmptyDataStore.hpp b/src/simplnx/DataStructure/EmptyDataStore.hpp index cbe280719e..259e3b303f 100644 --- a/src/simplnx/DataStructure/EmptyDataStore.hpp +++ b/src/simplnx/DataStructure/EmptyDataStore.hpp @@ -109,8 +109,11 @@ class EmptyDataStore : public AbstractDataStore } /** - * @brief Returns the store type e.g. in memory, out of core, etc. - * @return StoreType + * @brief Returns StoreType::Empty because this store is a metadata-only + * placeholder. The dataFormat() string records the intended storage + * strategy (e.g., "" for in-memory, or a named OOC format) so the + * framework knows what real store to create when execution begins. + * @return StoreType::Empty */ IDataStore::StoreType getStoreType() const override { @@ -118,8 +121,15 @@ class EmptyDataStore : public AbstractDataStore } /** - * @brief Checks and returns if the created data store should be in memory or handled out of core. - * @return bool + * @brief Returns the data format string that was specified at construction. + * + * This string indicates the intended storage strategy for the real data + * store that will replace this EmptyDataStore after preflight: + * - An empty string ("") means the data will be stored in-memory (DataStore). + * - A non-empty string names an out-of-core format (e.g., an OOC store + * implementation) that should be used for execution. + * + * @return std::string The data format identifier */ std::string dataFormat() const { @@ -158,11 +168,29 @@ class EmptyDataStore : public AbstractDataStore throw std::runtime_error("EmptyDataStore::setValue() is not implemented"); } + /** + * @brief Always throws because EmptyDataStore holds no data. EmptyDataStore + * is a metadata-only placeholder used during preflight; bulk data access is + * not supported. The store must be replaced with a real DataStore or OOC + * store before any data I/O is attempted. + * @param startIndex Unused + * @param buffer Unused + * @throw std::runtime_error Always + */ void copyIntoBuffer(usize startIndex, nonstd::span buffer) const override { throw std::runtime_error("EmptyDataStore::copyIntoBuffer() is not implemented"); } + /** + * @brief Always throws because EmptyDataStore holds no data. EmptyDataStore + * is a metadata-only placeholder used during preflight; bulk data access is + * not supported. The store must be replaced with a real DataStore or OOC + * store before any data I/O is attempted. + * @param startIndex Unused + * @param buffer Unused + * @throw std::runtime_error Always + */ void copyFromBuffer(usize startIndex, nonstd::span buffer) override { throw std::runtime_error("EmptyDataStore::copyFromBuffer() is not implemented"); diff --git a/src/simplnx/DataStructure/IDataStore.hpp b/src/simplnx/DataStructure/IDataStore.hpp index 1ac027835d..c32e54db3c 100644 --- a/src/simplnx/DataStructure/IDataStore.hpp +++ b/src/simplnx/DataStructure/IDataStore.hpp @@ -23,11 +23,35 @@ namespace nx::core class SIMPLNX_EXPORT IDataStore { public: + /** + * @brief Identifies how a data store manages its backing storage. + * + * Algorithms and I/O routines use this enum to determine whether data is + * immediately accessible in RAM or must be fetched from disk, and to + * distinguish real stores from preflight-only placeholders. + * + * - **InMemory** -- The store's data lives in a heap-allocated array that is + * always resident in RAM (DataStore). Element access via getValue/setValue + * and the bulk copyIntoBuffer/copyFromBuffer API are both cheap memory + * copies. + * + * - **OutOfCore** -- The store's data lives on disk in a chunked HDF5 dataset. + * Element access goes through chunk caching; the bulk copyIntoBuffer/ + * copyFromBuffer API translates flat ranges into efficient multi-chunk I/O. + * An earlier "EmptyOutOfCore" value was removed because the Empty type + * already covers placeholder semantics regardless of the eventual storage + * strategy. + * + * - **Empty** -- A metadata-only placeholder used during preflight + * (EmptyDataStore). Records tuple/component shape but holds no data. + * All data access methods throw. After preflight the Empty store is + * replaced with an InMemory or OutOfCore store before execution begins. + */ enum class StoreType : int32 { - InMemory = 0, - OutOfCore, - Empty + InMemory = 0, ///< Data is fully resident in a heap-allocated array (DataStore) + OutOfCore, ///< Data lives on disk in a chunked HDF5 dataset + Empty ///< Metadata-only placeholder used during preflight (EmptyDataStore) }; virtual ~IDataStore() = default; @@ -112,17 +136,27 @@ class SIMPLNX_EXPORT IDataStore /** * @brief Returns store-specific metadata needed for crash recovery. * - * The recovery file writer calls this to persist information about - * where this store's data lives, so it can be reattached after a - * crash. In-core stores return an empty map (their data is written - * directly to the recovery file). Out-of-core stores override this - * to return their backing file path, dataset path, chunk shape, etc. + * When the pipeline runner writes a recovery (.dream3d) file at + * the end of pipeline execution, it calls this method on every + * data store to capture whatever information is needed to reconnect + * the store to its data after a crash or unexpected termination. + * + * **In-memory stores (DataStore)** return an empty map because their + * data is written directly into the recovery file's HDF5 datasets; + * no extra metadata is required. + * + * **Out-of-core stores** override this to return key-value pairs + * describing their backing file path, HDF5 dataset path, chunk + * shape, and any other parameters needed to reconstruct the OOC + * store from the file on disk. * * Each key-value pair is written as an HDF5 string attribute on the - * array's dataset in the recovery file. The recovery loader reads - * these attributes to reconstruct the appropriate store. + * array's dataset inside the recovery file. The recovery loader + * reads these attributes to reconstruct the appropriate store + * subclass without loading the data into RAM. * - * @return std::map Key-value recovery metadata + * @return std::map Key-value pairs of + * recovery metadata. Empty for in-memory stores. */ virtual std::map getRecoveryMetadata() const { diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index e94faafba7..e76b2e32f3 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -83,8 +83,14 @@ std::unique_ptr DataIOCollection::createListStore(const std::string& return coreManager.listStoreCreationFnc(coreManager.formatName())(dataType, tupleShape); } +// --------------------------------------------------------------------------- +// Read-only reference DataStore creation (OOC DataArray support) +// --------------------------------------------------------------------------- + bool DataIOCollection::hasReadOnlyRefCreationFnc(const std::string& type) const { + // Search all registered IO managers for one that can create OOC DataStores + // for the requested format name. for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasReadOnlyRefCreationFnc(type)) @@ -98,6 +104,9 @@ bool DataIOCollection::hasReadOnlyRefCreationFnc(const std::string& type) const std::unique_ptr DataIOCollection::createReadOnlyRefStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& componentShape, const ShapeType& chunkShape) { + // Find the first IO manager that has a factory for the requested format and + // delegate the actual store construction to it. The factory will create a + // chunk-backed read-only store pointing at the HDF5 dataset. for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasReadOnlyRefCreationFnc(type)) @@ -106,11 +115,19 @@ std::unique_ptr DataIOCollection::createReadOnlyRefStore(const std:: return fnc(numericType, filePath, datasetPath, tupleShape, componentShape, chunkShape); } } + // No IO manager supports this format -- return nullptr so the caller can + // fall back to a different strategy (or report an error). return nullptr; } +// --------------------------------------------------------------------------- +// Read-only reference ListStore creation (OOC NeighborList support) +// --------------------------------------------------------------------------- + bool DataIOCollection::hasReadOnlyRefListCreationFnc(const std::string& type) const { + // Search all registered IO managers for one that can create OOC ListStores + // for the requested format name. for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasListStoreRefCreationFnc(type)) @@ -124,6 +141,8 @@ bool DataIOCollection::hasReadOnlyRefListCreationFnc(const std::string& type) co std::unique_ptr DataIOCollection::createReadOnlyRefListStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& chunkShape) { + // Find the first IO manager that has a ListStore reference factory for the + // requested format and delegate construction. for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasListStoreRefCreationFnc(type)) @@ -132,11 +151,18 @@ std::unique_ptr DataIOCollection::createReadOnlyRefListStore(const s return fnc(numericType, filePath, datasetPath, tupleShape, chunkShape); } } + // No IO manager supports this format for list stores. return nullptr; } +// --------------------------------------------------------------------------- +// StringStore creation +// --------------------------------------------------------------------------- + bool DataIOCollection::hasStringStoreCreationFnc(const std::string& type) const { + // Search all registered IO managers for one that provides a StringStore + // factory for the requested format name. for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasStringStoreCreationFnc(type)) @@ -149,6 +175,8 @@ bool DataIOCollection::hasStringStoreCreationFnc(const std::string& type) const std::unique_ptr DataIOCollection::createStringStore(const std::string& type, const ShapeType& tupleShape) { + // Find the first IO manager that has a StringStore factory for the requested + // format and delegate construction. for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasStringStoreCreationFnc(type)) @@ -157,11 +185,19 @@ std::unique_ptr DataIOCollection::createStringStore(const s return fnc(tupleShape); } } + // No IO manager supports this format for string stores. return nullptr; } +// --------------------------------------------------------------------------- +// Post-pipeline store finalization +// --------------------------------------------------------------------------- + void DataIOCollection::finalizeStores(DataStructure& dataStructure) { + // Iterate over every registered IO manager and invoke its finalize callback + // if one exists. Each callback transitions its own stores from write mode to + // read-only mode (e.g., closing HDF5 write handles, opening read handles). for(const auto& [ioType, ioManager] : m_ManagerMap) { if(ioManager->hasFinalizeStoresFnc(ioType)) @@ -171,8 +207,13 @@ void DataIOCollection::finalizeStores(DataStructure& dataStructure) } } +// --------------------------------------------------------------------------- +// Format resolver hook (decides in-memory vs OOC for new arrays) +// --------------------------------------------------------------------------- + void DataIOCollection::setFormatResolver(FormatResolverFnc resolver) { + // Store the callback. An empty std::function effectively disables the resolver. m_FormatResolver = std::move(resolver); } @@ -183,15 +224,23 @@ bool DataIOCollection::hasFormatResolver() const std::string DataIOCollection::resolveFormat(DataType numericType, const ShapeType& tupleShape, const ShapeType& componentShape, uint64 dataSizeBytes) const { + // If no resolver is installed, return "" to signal the default in-memory format. if(!m_FormatResolver) { return ""; } + // Delegate to the plugin-provided resolver. It returns a format name like + // "HDF5-OOC" for large arrays, or "" to keep the default in-memory store. return m_FormatResolver(numericType, tupleShape, componentShape, dataSizeBytes); } +// --------------------------------------------------------------------------- +// Backfill handler hook (replaces placeholders after .dream3d import) +// --------------------------------------------------------------------------- + void DataIOCollection::setBackfillHandler(BackfillHandlerFnc handler) { + // Store the callback. An empty std::function effectively disables backfill. m_BackfillHandler = std::move(handler); } @@ -202,10 +251,15 @@ bool DataIOCollection::hasBackfillHandler() const Result<> DataIOCollection::runBackfillHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader) { + // If no handler is installed, this is the in-memory code path: placeholders + // will be replaced elsewhere (or eagerly loaded). Return success. if(!m_BackfillHandler) { return {}; } + // Delegate to the plugin-provided handler, which replaces placeholder stores + // (EmptyDataStore, EmptyListStore, EmptyStringStore) with real stores + // (typically OOC read-only references pointing at the same HDF5 file). return m_BackfillHandler(dataStructure, paths, fileReader); } @@ -241,8 +295,14 @@ DataIOCollection::const_iterator DataIOCollection::end() const return m_ManagerMap.end(); } +// --------------------------------------------------------------------------- +// Write-array-override hook (intercepts DataObject writes during HDF5 save) +// --------------------------------------------------------------------------- + void DataIOCollection::setWriteArrayOverride(WriteArrayOverrideFnc fnc) { + // Store the callback. Registration alone does not activate the override; + // setWriteArrayOverrideActive(true) or WriteArrayOverrideGuard is also needed. m_WriteArrayOverride = std::move(fnc); } @@ -253,6 +313,8 @@ bool DataIOCollection::hasWriteArrayOverride() const void DataIOCollection::setWriteArrayOverrideActive(bool active) { + // Toggle the gate flag. The override fires only when both the callback is + // registered (m_WriteArrayOverride is non-empty) AND this flag is true. m_WriteArrayOverrideActive = active; } @@ -263,10 +325,16 @@ bool DataIOCollection::isWriteArrayOverrideActive() const std::optional> DataIOCollection::runWriteArrayOverride(HDF5::DataStructureWriter& writer, const DataObject* dataObject, HDF5::GroupIO& parentGroup) { + // Short-circuit: if the override is not active or not registered, return + // std::nullopt so the caller proceeds with the normal HDF5 write path. if(!m_WriteArrayOverrideActive || !m_WriteArrayOverride) { return std::nullopt; } + // Delegate to the plugin-provided callback. It may return: + // - std::nullopt => decline this object, caller should use normal write + // - Result<>{} => object was written successfully by the override + // - Result<> error => override attempted to write but failed return m_WriteArrayOverride(writer, dataObject, parentGroup); } } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index 46092aded9..00238d8eb7 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -44,108 +44,202 @@ class SIMPLNX_EXPORT DataIOCollection using const_iterator = typename map_type::const_iterator; /** - * @brief A callback that decides the storage format for an array given its metadata. - * Returns a format string (e.g., "HDF5-OOC") or "" for default in-memory storage. - * Plugins register resolvers to control format decisions without core knowing about specific formats. + * @brief Callback that decides which storage format to use when a new array is + * created during filter execution (e.g., in CreateArrayAction). + * + * The resolver inspects the array's metadata (numeric type, shape, total byte + * size) and returns either: + * - A format name string (e.g., "HDF5-OOC") to request out-of-core storage, or + * - An empty string "" to use the default in-memory DataStore. + * + * This hook allows the OOC plugin to redirect large arrays to disk-backed + * storage without the core library needing to know about any specific OOC + * format. Only one resolver may be active at a time. + * + * @param numericType The element data type (float32, int32, uint8, etc.) + * @param tupleShape The tuple dimensions of the array + * @param componentShape The component dimensions per tuple + * @param dataSizeBytes The total size of the array data in bytes + * @return Format name string, or empty string for in-memory default */ using FormatResolverFnc = std::function; /** - * @brief A callback that takes over post-import finalization of placeholder stores. - * Called once after .dream3d paths are imported with placeholder stores (EmptyDataStore, - * EmptyListStore, EmptyStringStore). The plugin is responsible for replacing placeholders - * with real stores (e.g., OOC references, or eager-loaded data). + * @brief Callback invoked once after a .dream3d file has been imported with + * placeholder (empty) stores. + * + * During import, the HDF5 reader creates lightweight placeholder stores + * (EmptyDataStore, EmptyListStore, EmptyStringStore) for every array so that + * the DataStructure's topology is complete without loading data. The backfill + * handler is then called with the list of imported DataPaths and the still-open + * HDF5 file reader, giving the OOC plugin the opportunity to replace each + * placeholder with a real store -- typically a read-only OOC reference that + * lazily reads chunks from the same file. + * + * Only one backfill handler may be registered at a time. If no handler is + * registered, the placeholders remain (which is valid for in-memory workflows + * that eagerly load data through a different path). + * + * @param dataStructure The DataStructure containing placeholder stores to backfill + * @param paths DataPaths of the objects imported from the file + * @param fileReader Open HDF5 file reader for the source .dream3d file + * @return Result<> indicating success or describing any errors */ using BackfillHandlerFnc = std::function(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader)>; /** - * @brief A callback that can override how a DataObject is written to HDF5. - * Used by plugins to intercept specific objects during recovery writes. - * Returns std::nullopt to pass through to the normal write path. - * Returns Result<> to indicate the hook handled the write. + * @brief Callback that can intercept and override how a DataObject is written + * to an HDF5 file during a DataStructure save operation. + * + * This hook exists primarily for recovery-file writes in OOC mode. When the + * OOC plugin is active, arrays may be backed by read-only references to an + * existing HDF5 file. During a recovery save, those arrays should not be + * re-serialized from memory (they may not even be fully loaded); instead the + * override callback can create an HDF5 hard link or external link back to the + * original dataset. + * + * The callback uses a tri-state return: + * - std::nullopt -- the hook declines to handle this object; the normal + * write path should proceed. + * - Result<>{} -- the hook successfully wrote the object; the normal + * write path should be skipped. + * - Result<> with errors -- the hook attempted to write but failed. + * + * The override is only invoked when both registered AND activated via + * setWriteArrayOverrideActive(true) or WriteArrayOverrideGuard. This two-step + * design lets the plugin register the callback once at load time while only + * activating it during specific save operations. + * + * @param writer The DataStructureWriter managing the save operation + * @param dataObject The DataObject about to be written + * @param parentGroup The HDF5 group that would normally receive the object + * @return std::nullopt to fall through, or Result<> to indicate the hook handled (or failed) the write */ using WriteArrayOverrideFnc = std::function>(HDF5::DataStructureWriter& writer, const DataObject* dataObject, HDF5::GroupIO& parentGroup)>; /** - * @brief Sets the format resolver callback. Only one resolver can be active at a time. - * Passing nullptr disables the resolver (resolveFormat() will return ""). - * @param resolver The resolver callback (or nullptr to unset) + * @brief Registers or clears the format resolver callback. + * + * Only one resolver can be active at a time. The OOC plugin typically calls + * this once during plugin loading to install its size-threshold logic. Passing + * a default-constructed (empty) std::function disables the resolver so that + * resolveFormat() returns "" for all queries. + * + * @param resolver The resolver callback, or an empty std::function to clear it */ void setFormatResolver(FormatResolverFnc resolver); /** - * @brief Checks whether a format resolver is registered. - * @return true if a resolver callback is currently set + * @brief Checks whether a format resolver callback is currently registered. + * @return true if a non-empty resolver callback is set */ bool hasFormatResolver() const; /** - * @brief Asks the registered resolver what format to use for the given array. - * Returns "" if no resolver is registered or if the resolver decides default/in-memory. - * @param numericType The numeric type of the array - * @param tupleShape The tuple dimensions + * @brief Queries the registered format resolver to determine which storage + * format should be used for a new array with the given metadata. + * + * This method is called by CreateArrayAction and similar actions when they + * allocate a new DataStore. If a resolver is registered, it is invoked with + * the array's properties and its return value selects the format. If no + * resolver is registered, or the resolver returns "", the caller falls back + * to the default in-memory DataStore. + * + * @param numericType The element data type of the array + * @param tupleShape The tuple dimensions * @param componentShape The component dimensions - * @param dataSizeBytes The total data size in bytes - * @return Format name string (e.g., "HDF5-OOC") or empty string + * @param dataSizeBytes The total size of the array data in bytes + * @return A format name string (e.g., "HDF5-OOC") to use a specific format, + * or an empty string for the default in-memory format */ std::string resolveFormat(DataType numericType, const ShapeType& tupleShape, const ShapeType& componentShape, uint64 dataSizeBytes) const; /** - * @brief Sets the backfill handler. Passing nullptr disables the handler. - * @param handler The backfill handler callback + * @brief Registers or clears the backfill handler callback. + * + * The OOC plugin calls this once during loading to install a handler that + * replaces placeholder stores with OOC reference stores after a .dream3d + * file is imported. Passing an empty std::function disables the handler. + * + * @param handler The backfill handler callback, or an empty std::function to clear it */ void setBackfillHandler(BackfillHandlerFnc handler); /** - * @brief Checks whether a backfill handler is registered. - * @return true if a handler is currently set + * @brief Checks whether a backfill handler callback is currently registered. + * @return true if a non-empty handler callback is set */ bool hasBackfillHandler() const; /** - * @brief Runs the registered backfill handler. Returns an empty success Result if no handler is set. - * @param dataStructure The DataStructure containing placeholder stores - * @param paths Paths imported from the file that need finalization - * @param fileReader Open HDF5 file reader for the source file - * @return Result indicating success or failure + * @brief Invokes the registered backfill handler to replace placeholder stores + * with real stores after importing a .dream3d file. + * + * If no handler is registered, this is a no-op and returns an empty success + * Result. This allows the in-memory code path to skip backfill entirely while + * the OOC code path processes every imported array. + * + * @param dataStructure The DataStructure whose placeholder stores should be replaced + * @param paths DataPaths of the objects imported from the file + * @param fileReader Open HDF5 file reader for the source .dream3d file + * @return Result<> indicating success, or containing errors from the handler */ Result<> runBackfillHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader); /** - * @brief Sets the write-array-override callback. Passing nullptr disables the override. - * The override is registered at plugin load time but only fires when activated - * via setWriteArrayOverrideActive(true). - * @param fnc The override callback (or nullptr to unset) + * @brief Registers or clears the write-array-override callback. + * + * The OOC plugin registers this once at load time. The callback is only + * invoked during DataStructure writes when it is also activated (see + * setWriteArrayOverrideActive()). This separation lets the plugin install + * the hook eagerly while only enabling it for specific save operations + * (e.g., recovery files) where OOC arrays need special handling. + * + * @param fnc The override callback, or an empty std::function to clear it */ void setWriteArrayOverride(WriteArrayOverrideFnc fnc); /** - * @brief Checks whether a write-array-override callback is registered. - * @return true if an override callback is currently set + * @brief Checks whether a write-array-override callback is currently registered. + * @return true if a non-empty override callback is set */ bool hasWriteArrayOverride() const; /** - * @brief Activates or deactivates the write-array-override. The override only - * fires when both registered AND active. Use WriteArrayOverrideGuard for RAII. - * @param active Whether the override should be active + * @brief Activates or deactivates the write-array-override. + * + * The override is invoked only when both registered AND active. Prefer using + * WriteArrayOverrideGuard for scoped activation to ensure the override is + * always deactivated when the save operation completes (even on exceptions). + * + * @param active true to activate, false to deactivate */ void setWriteArrayOverrideActive(bool active); /** * @brief Checks whether the write-array-override is currently active. - * @return true if the override is active + * + * An override can be registered but inactive (the default state after + * setWriteArrayOverride()). It must be explicitly activated before it fires. + * + * @return true if the override is currently active */ bool isWriteArrayOverrideActive() const; /** - * @brief Runs the write-array-override if registered and active. - * Returns std::nullopt if no override is registered, not active, or if the - * override passes on this object. - * @param writer The DataStructureWriter - * @param dataObject The DataObject being written - * @param parentGroup The HDF5 parent group - * @return std::optional> The override result, or std::nullopt to fall through + * @brief Invokes the write-array-override callback for a single DataObject + * if the override is both registered and active. + * + * Called by DataStructureWriter for each DataObject during a save. The + * override can choose to handle the write (returning Result<>), or decline + * by returning std::nullopt, in which case the normal HDF5 write path + * proceeds. + * + * @param writer The DataStructureWriter performing the save + * @param dataObject The DataObject about to be written + * @param parentGroup The HDF5 group that would normally receive the object + * @return std::nullopt if the override is not active, not registered, or + * declines this object; otherwise Result<> from the override */ std::optional> runWriteArrayOverride(HDF5::DataStructureWriter& writer, const DataObject* dataObject, HDF5::GroupIO& parentGroup); @@ -231,65 +325,98 @@ class SIMPLNX_EXPORT DataIOCollection } /** - * @brief Checks if any registered IO manager can create read-only reference stores. - * @param type The data format name (e.g. "HDF5-OOC") - * @return true if a read-only reference creation function is registered + * @brief Checks whether any registered IO manager provides a factory for + * creating read-only reference DataStores (OOC stores backed by an existing + * HDF5 dataset). + * + * @param type The format name to query (e.g., "HDF5-OOC") + * @return true if at least one registered IO manager has a factory for @p type */ bool hasReadOnlyRefCreationFnc(const std::string& type) const; /** - * @brief Creates a read-only data store that references an existing dataset in an HDF5 file. - * @param type The data format name (e.g. "HDF5-OOC") - * @param numericType The numeric data type - * @param filePath Path to the existing HDF5 file - * @param datasetPath HDF5-internal path to the dataset - * @param tupleShape Tuple dimensions - * @param componentShape Component dimensions - * @param chunkShape Chunk dimensions for the logical cache - * @return A new read-only IDataStore, or nullptr if no factory is registered + * @brief Creates a read-only DataStore that lazily reads data from an existing + * dataset inside an HDF5 file. + * + * This is the primary entry point for constructing OOC DataArray backing + * stores. The method searches all registered IO managers for one that handles + * the requested format, then delegates to its DataStoreRefCreateFnc factory. + * The resulting store does not load the full dataset into memory; instead it + * reads chunks on demand using the provided chunk shape. + * + * @param type The format name (e.g., "HDF5-OOC") + * @param numericType The element data type (float32, int32, etc.) + * @param filePath Filesystem path to the HDF5 file containing the data + * @param datasetPath HDF5-internal path to the dataset + * @param tupleShape Logical tuple dimensions of the array + * @param componentShape Component dimensions per tuple + * @param chunkShape Chunk dimensions controlling how many tuples are loaded per I/O operation + * @return A new read-only IDataStore, or nullptr if no factory handles @p type */ std::unique_ptr createReadOnlyRefStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& componentShape, const ShapeType& chunkShape); /** - * @brief Checks if any registered IO manager can create read-only reference list stores. - * @param type The format name (e.g. "HDF5-OOC") - * @return true if a registered factory exists + * @brief Checks whether any registered IO manager provides a factory for + * creating read-only reference ListStores (OOC NeighborList stores backed by + * existing HDF5 datasets). + * + * @param type The format name to query (e.g., "HDF5-OOC") + * @return true if at least one registered IO manager has a factory for @p type */ bool hasReadOnlyRefListCreationFnc(const std::string& type) const; /** - * @brief Creates a read-only list store that references existing datasets in an HDF5 file. - * @param type The format name (e.g. "HDF5-OOC") - * @param numericType The numeric data type - * @param filePath Path to the existing HDF5 file + * @brief Creates a read-only ListStore that lazily reads variable-length + * NeighborList data from an existing dataset inside an HDF5 file. + * + * Analogous to createReadOnlyRefStore() but for NeighborList arrays. The + * resulting store reads list entries on demand using chunk-based I/O. + * + * @param type The format name (e.g., "HDF5-OOC") + * @param numericType The element data type + * @param filePath Filesystem path to the HDF5 file * @param datasetPath HDF5-internal path to the dataset - * @param tupleShape Tuple dimensions - * @param chunkShape Chunk dimensions for the logical cache (tuples per chunk) - * @return A new read-only IListStore, or nullptr if no factory is registered + * @param tupleShape Logical tuple dimensions + * @param chunkShape Chunk dimensions (tuples per chunk) + * @return A new read-only IListStore, or nullptr if no factory handles @p type */ std::unique_ptr createReadOnlyRefListStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& chunkShape); /** - * @brief Checks if a string store creation function exists for the specified type. - * @param type The format name - * @return bool True if a registered factory exists + * @brief Checks whether any registered IO manager provides a factory for + * creating StringStores of the specified format. + * + * @param type The format name to query + * @return true if at least one registered IO manager has a StringStore factory for @p type */ bool hasStringStoreCreationFnc(const std::string& type) const; /** - * @brief Creates a string store of the specified type. - * @param type The format name - * @param tupleShape Tuple dimensions - * @return A new AbstractStringStore, or nullptr if no factory is registered + * @brief Creates a StringStore (backing store for StringArray) of the + * specified format and dimensions. + * + * Searches all registered IO managers for one that handles the requested + * format, then delegates to its StringStoreCreateFnc factory. The resulting + * store may be in-memory or disk-backed depending on the IO manager. + * + * @param type The format name to use + * @param tupleShape Tuple dimensions for the string array + * @return A new AbstractStringStore, or nullptr if no factory handles @p type */ std::unique_ptr createStringStore(const std::string& type, const ShapeType& tupleShape); /** - * @brief Calls all registered finalize-stores callbacks on the DataStructure. - * Called after pipeline execution to prepare stores for reading (e.g., close - * write handles and switch to read-only mode). No-op if no callbacks registered. + * @brief Invokes all registered finalize-stores callbacks across every IO + * manager. + * + * Called after a pipeline finishes executing. Each IO manager that registered + * a FinalizeStoresFnc gets a chance to walk the DataStructure and transition + * its stores from write mode to read-only mode (e.g., closing HDF5 write + * handles and re-opening as read handles). This is a no-op if no IO manager + * has registered a finalize callback. + * * @param dataStructure The DataStructure whose stores should be finalized */ void finalizeStores(DataStructure& dataStructure); @@ -320,42 +447,80 @@ class SIMPLNX_EXPORT DataIOCollection private: map_type m_ManagerMap; - FormatResolverFnc m_FormatResolver; - BackfillHandlerFnc m_BackfillHandler; - WriteArrayOverrideFnc m_WriteArrayOverride; - bool m_WriteArrayOverrideActive = false; + FormatResolverFnc m_FormatResolver; ///< Plugin-provided callback that selects storage format for new arrays + BackfillHandlerFnc m_BackfillHandler; ///< Plugin-provided callback that replaces placeholders after .dream3d import + WriteArrayOverrideFnc m_WriteArrayOverride; ///< Plugin-provided callback that intercepts DataObject writes to HDF5 + bool m_WriteArrayOverrideActive = false; ///< Gate flag: override fires only when both registered and active }; /** - * @brief RAII guard that activates the write-array-override for its lifetime. - * On construction, sets the override active. On destruction, deactivates it. + * @brief RAII guard that activates the write-array-override callback for the + * duration of a scoped block, then automatically deactivates it on destruction. + * + * The write-array-override in DataIOCollection uses a two-phase design: + * 1. The callback is **registered** once at plugin load time via + * DataIOCollection::setWriteArrayOverride(). + * 2. The callback is **activated** only for specific save operations (e.g., + * recovery file writes) where OOC arrays need special write handling. + * + * This guard manages phase 2. Constructing the guard calls + * setWriteArrayOverrideActive(true), and the destructor calls + * setWriteArrayOverrideActive(false). This guarantees the override is + * deactivated even if an exception occurs during the save. + * + * Usage: + * @code + * { + * WriteArrayOverrideGuard guard(ioCollection); + * // All DataStructureWriter::writeDataObject() calls in this scope will + * // consult the override callback before performing the default HDF5 write. + * writer.writeDataStructure(dataStructure); + * } + * // Override is now deactivated; normal writes proceed without consulting the callback. + * @endcode + * + * This class is non-copyable and non-movable to prevent accidental misuse + * (e.g., moving the guard out of scope prematurely). */ class SIMPLNX_EXPORT WriteArrayOverrideGuard { public: + /** + * @brief Constructs the guard and activates the write-array-override. + * @param ioCollection The DataIOCollection whose override should be activated. + * May be nullptr, in which case this guard is a no-op. + */ explicit WriteArrayOverrideGuard(std::shared_ptr ioCollection) : m_IOCollection(std::move(ioCollection)) { + // Activate the override so DataStructureWriter will consult the callback. + // Null-check allows callers to conditionally construct the guard. if(m_IOCollection) { m_IOCollection->setWriteArrayOverrideActive(true); } } + /** + * @brief Destructor that deactivates the write-array-override, restoring + * normal write behavior. + */ ~WriteArrayOverrideGuard() { + // Deactivate the override so subsequent writes go through the normal path. if(m_IOCollection) { m_IOCollection->setWriteArrayOverrideActive(false); } } + // Non-copyable and non-movable to prevent scope-escape bugs. WriteArrayOverrideGuard(const WriteArrayOverrideGuard&) = delete; WriteArrayOverrideGuard& operator=(const WriteArrayOverrideGuard&) = delete; WriteArrayOverrideGuard(WriteArrayOverrideGuard&&) = delete; WriteArrayOverrideGuard& operator=(WriteArrayOverrideGuard&&) = delete; private: - std::shared_ptr m_IOCollection; + std::shared_ptr m_IOCollection; ///< Held alive for the guard's lifetime to ensure safe deactivation }; } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp index bc409542f5..06928ab394 100644 --- a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp +++ b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.cpp @@ -56,6 +56,10 @@ void IDataIOManager::addListStoreCreationFnc(const std::string& type, ListStoreC m_ListStoreCreationMap[type] = creationFnc; } +// --------------------------------------------------------------------------- +// Read-only reference DataStore factory (OOC DataArray support) +// --------------------------------------------------------------------------- + bool IDataIOManager::hasReadOnlyRefCreationFnc(const std::string& type) const { return m_ReadOnlyRefCreationMap.find(type) != m_ReadOnlyRefCreationMap.end(); @@ -63,6 +67,8 @@ bool IDataIOManager::hasReadOnlyRefCreationFnc(const std::string& type) const IDataIOManager::DataStoreRefCreateFnc IDataIOManager::readOnlyRefCreationFnc(const std::string& type) const { + // Look up the factory by format name; return nullptr if this manager does + // not provide OOC DataStore support for the requested format. auto iter = m_ReadOnlyRefCreationMap.find(type); if(iter == m_ReadOnlyRefCreationMap.end()) { @@ -73,9 +79,14 @@ IDataIOManager::DataStoreRefCreateFnc IDataIOManager::readOnlyRefCreationFnc(con void IDataIOManager::addReadOnlyRefCreationFnc(const std::string& type, DataStoreRefCreateFnc creationFnc) { + // Register (or replace) the OOC DataStore factory for this format name. m_ReadOnlyRefCreationMap[type] = std::move(creationFnc); } +// --------------------------------------------------------------------------- +// Read-only reference ListStore factory (OOC NeighborList support) +// --------------------------------------------------------------------------- + bool IDataIOManager::hasListStoreRefCreationFnc(const std::string& type) const { return m_ListStoreRefCreationMap.find(type) != m_ListStoreRefCreationMap.cend(); @@ -83,6 +94,8 @@ bool IDataIOManager::hasListStoreRefCreationFnc(const std::string& type) const IDataIOManager::ListStoreRefCreateFnc IDataIOManager::listStoreRefCreationFnc(const std::string& type) const { + // Look up the factory by format name; return nullptr if this manager does + // not provide OOC ListStore support for the requested format. auto iter = m_ListStoreRefCreationMap.find(type); if(iter == m_ListStoreRefCreationMap.cend()) { @@ -93,9 +106,14 @@ IDataIOManager::ListStoreRefCreateFnc IDataIOManager::listStoreRefCreationFnc(co void IDataIOManager::addListStoreRefCreationFnc(const std::string& type, ListStoreRefCreateFnc creationFnc) { + // Register (or replace) the OOC ListStore factory for this format name. m_ListStoreRefCreationMap[type] = std::move(creationFnc); } +// --------------------------------------------------------------------------- +// StringStore factory (StringArray backing store) +// --------------------------------------------------------------------------- + bool IDataIOManager::hasStringStoreCreationFnc(const std::string& type) const { return m_StringStoreCreationMap.find(type) != m_StringStoreCreationMap.cend(); @@ -103,6 +121,8 @@ bool IDataIOManager::hasStringStoreCreationFnc(const std::string& type) const IDataIOManager::StringStoreCreateFnc IDataIOManager::stringStoreCreationFnc(const std::string& type) const { + // Look up the factory by format name; return nullptr if this manager does + // not provide StringStore support for the requested format. auto iter = m_StringStoreCreationMap.find(type); if(iter == m_StringStoreCreationMap.cend()) { @@ -113,9 +133,14 @@ IDataIOManager::StringStoreCreateFnc IDataIOManager::stringStoreCreationFnc(cons void IDataIOManager::addStringStoreCreationFnc(const std::string& type, StringStoreCreateFnc creationFnc) { + // Register (or replace) the StringStore factory for this format name. m_StringStoreCreationMap[type] = std::move(creationFnc); } +// --------------------------------------------------------------------------- +// Post-pipeline store finalization callbacks +// --------------------------------------------------------------------------- + bool IDataIOManager::hasFinalizeStoresFnc(const std::string& type) const { return m_FinalizeStoresMap.find(type) != m_FinalizeStoresMap.end(); @@ -123,6 +148,7 @@ bool IDataIOManager::hasFinalizeStoresFnc(const std::string& type) const IDataIOManager::FinalizeStoresFnc IDataIOManager::finalizeStoresFnc(const std::string& type) const { + // Look up the finalize callback; return nullptr if none is registered. auto iter = m_FinalizeStoresMap.find(type); if(iter == m_FinalizeStoresMap.end()) { @@ -133,6 +159,7 @@ IDataIOManager::FinalizeStoresFnc IDataIOManager::finalizeStoresFnc(const std::s void IDataIOManager::addFinalizeStoresFnc(const std::string& type, FinalizeStoresFnc fnc) { + // Register (or replace) the post-pipeline finalization callback for this format. m_FinalizeStoresMap[type] = std::move(fnc); } } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp index de0dd0d15b..6d3d210c81 100644 --- a/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp +++ b/src/simplnx/DataStructure/IO/Generic/IDataIOManager.hpp @@ -30,20 +30,81 @@ class SIMPLNX_EXPORT IDataIOManager using factory_id_type = std::string; using factory_ptr = std::shared_ptr; using factory_collection = std::map; + /** + * @brief Factory callback for creating a new in-memory DataStore. + * + * Takes the numeric type, tuple shape, component shape, and an optional chunk + * shape hint. Returns a newly allocated IDataStore. Registered by IO managers + * that provide writable storage (e.g., CoreDataIOManager for in-memory, or an + * OOC manager for chunk-backed stores). + */ using DataStoreCreateFnc = std::function(DataType, const ShapeType&, const ShapeType&, const std::optional&)>; + + /** + * @brief Factory callback for creating a new in-memory NeighborList store. + * + * Takes the numeric type and tuple shape. Returns a newly allocated IListStore. + * Used by IO managers that can provide writable list-based storage. + */ using ListStoreCreateFnc = std::function(DataType, const ShapeType&)>; + + /** + * @brief Factory callback for creating a read-only DataStore that references + * an existing dataset inside an HDF5 file without loading it into memory. + * + * This is the core factory for out-of-core (OOC) DataArray access. The + * resulting store lazily reads data from disk on demand via chunk-based I/O. + * + * @param numericType The element data type (float32, int32, etc.) + * @param filePath Filesystem path to the HDF5 file containing the data + * @param datasetPath HDF5-internal path to the dataset (e.g., "/DataContainer/CellData/Phases") + * @param tupleShape Logical tuple dimensions of the array + * @param componentShape Component dimensions per tuple + * @param chunkShape Chunk dimensions controlling how many tuples are loaded per I/O operation + */ using DataStoreRefCreateFnc = std::function(DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& componentShape, const ShapeType& chunkShape)>; + + /** + * @brief Factory callback for creating a read-only NeighborList store that + * references existing datasets inside an HDF5 file. + * + * Analogous to DataStoreRefCreateFnc but for NeighborList data. The resulting + * store lazily reads variable-length lists from disk on demand. + * + * @param numericType The element data type + * @param filePath Filesystem path to the HDF5 file + * @param datasetPath HDF5-internal path to the dataset + * @param tupleShape Logical tuple dimensions + * @param chunkShape Chunk dimensions (tuples per chunk) + */ using ListStoreRefCreateFnc = std::function(DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& chunkShape)>; + + /** + * @brief Factory callback for creating a new StringStore (e.g., for StringArray). + * + * Takes the tuple shape and returns a newly allocated AbstractStringStore. + * Registered by IO managers that support string storage (in-memory or OOC). + */ using StringStoreCreateFnc = std::function(const ShapeType& tupleShape)>; + + /** + * @brief Post-pipeline callback that finalizes all stores owned by a given + * IO format. + * + * Called after a pipeline finishes executing. The callback walks the + * DataStructure and transitions stores from write mode to read-only mode + * (e.g., closing HDF5 write handles and opening read handles). This ensures + * that subsequent reads from the stores see the fully written data. + */ using FinalizeStoresFnc = std::function; - using DataStoreCreationMap = std::map; - using ListStoreCreationMap = std::map; - using DataStoreRefCreationMap = std::map; - using ListStoreRefCreationMap = std::map; - using StringStoreCreationMap = std::map; - using FinalizeStoresMap = std::map; + using DataStoreCreationMap = std::map; ///< Maps format name -> writable DataStore factory + using ListStoreCreationMap = std::map; ///< Maps format name -> writable ListStore factory + using DataStoreRefCreationMap = std::map; ///< Maps format name -> read-only DataStore factory (OOC) + using ListStoreRefCreationMap = std::map; ///< Maps format name -> read-only ListStore factory (OOC) + using StringStoreCreationMap = std::map; ///< Maps format name -> StringStore factory + using FinalizeStoresMap = std::map; ///< Maps format name -> post-pipeline finalize callback virtual ~IDataIOManager() noexcept; @@ -124,58 +185,89 @@ class SIMPLNX_EXPORT IDataIOManager ListStoreCreateFnc listStoreCreationFnc(const std::string& type) const; /** - * @brief Checks if a read-only reference store creation function exists for the specified type. - * @param type The data store type name - * @return bool True if the creation function exists, false otherwise + * @brief Checks whether this IO manager has registered a factory for creating + * read-only reference DataStores (out-of-core stores that lazily read from an + * existing HDF5 file). + * + * Callers should check this before calling readOnlyRefCreationFnc() to avoid + * a null return. + * + * @param type The format name to look up (e.g., "HDF5-OOC") + * @return true if a DataStoreRefCreateFnc is registered for @p type */ bool hasReadOnlyRefCreationFnc(const std::string& type) const; /** - * @brief Returns the read-only reference store creation function for the specified type. - * @param type The data store type name - * @return DataStoreRefCreateFnc The creation function, or nullptr if not found + * @brief Returns the factory callback for creating a read-only reference + * DataStore that performs lazy chunk-based reads from an HDF5 file. + * + * The returned callback, when invoked, produces a store that does not copy + * the full dataset into memory; instead it reads chunks on demand. + * + * @param type The format name to look up (e.g., "HDF5-OOC") + * @return The registered DataStoreRefCreateFnc, or nullptr if none is registered */ DataStoreRefCreateFnc readOnlyRefCreationFnc(const std::string& type) const; /** - * @brief Checks if a read-only reference list store creation function exists for the specified type. - * @param type The format name - * @return bool True if the creation function exists, false otherwise + * @brief Checks whether this IO manager has registered a factory for creating + * read-only reference ListStores (out-of-core NeighborList stores). + * + * @param type The format name to look up (e.g., "HDF5-OOC") + * @return true if a ListStoreRefCreateFnc is registered for @p type */ bool hasListStoreRefCreationFnc(const std::string& type) const; /** - * @brief Returns the read-only reference list store creation function for the specified type. - * @param type The format name - * @return ListStoreRefCreateFnc The creation function, or nullptr if not found + * @brief Returns the factory callback for creating a read-only reference + * ListStore that performs lazy reads of variable-length NeighborList data + * from an HDF5 file. + * + * @param type The format name to look up (e.g., "HDF5-OOC") + * @return The registered ListStoreRefCreateFnc, or nullptr if none is registered */ ListStoreRefCreateFnc listStoreRefCreationFnc(const std::string& type) const; /** - * @brief Checks if a string store creation function exists for the specified type. - * @param type The format name - * @return bool True if the creation function exists, false otherwise + * @brief Checks whether this IO manager has registered a factory for creating + * StringStores (stores backing StringArray objects). + * + * @param type The format name to look up + * @return true if a StringStoreCreateFnc is registered for @p type */ bool hasStringStoreCreationFnc(const std::string& type) const; /** - * @brief Returns the string store creation function for the specified type. - * @param type The format name - * @return StringStoreCreateFnc The creation function, or nullptr if not found + * @brief Returns the factory callback for creating a StringStore. The + * resulting store backs a StringArray and may be in-memory or out-of-core + * depending on the IO manager that registered it. + * + * @param type The format name to look up + * @return The registered StringStoreCreateFnc, or nullptr if none is registered */ StringStoreCreateFnc stringStoreCreationFnc(const std::string& type) const; /** - * @brief Checks if a finalize-stores callback exists for the specified type. - * @param type The data store type name - * @return bool True if the callback exists, false otherwise + * @brief Checks whether this IO manager has registered a finalize-stores + * callback for the specified format. + * + * The finalize callback is invoked after pipeline execution to transition + * stores from write mode to read-only mode. + * + * @param type The format name to look up + * @return true if a FinalizeStoresFnc is registered for @p type */ bool hasFinalizeStoresFnc(const std::string& type) const; /** - * @brief Returns the finalize-stores callback for the specified type. - * @param type The data store type name - * @return FinalizeStoresFnc The callback, or nullptr if not found + * @brief Returns the finalize-stores callback for the specified format. + * + * When invoked, this callback walks the DataStructure and closes any write + * handles, replacing them with read-only handles so that data written during + * the pipeline is accessible for subsequent reads. + * + * @param type The format name to look up + * @return The registered FinalizeStoresFnc, or nullptr if none is registered */ FinalizeStoresFnc finalizeStoresFnc(const std::string& type) const; @@ -200,32 +292,58 @@ class SIMPLNX_EXPORT IDataIOManager void addListStoreCreationFnc(const std::string& type, ListStoreCreateFnc creationFnc); /** - * @brief Adds a read-only reference store creation function for the specified type. - * @param type The data store type name - * @param creationFnc The creation function to add + * @brief Registers a factory callback that creates read-only reference + * DataStores for the given format name. + * + * Derived IO managers call this during construction to advertise their + * ability to create OOC DataStores. DataIOCollection::createReadOnlyRefStore() + * dispatches to the callback registered here. + * + * @param type The format name to register under (e.g., "HDF5-OOC") + * @param creationFnc The factory callback. Replaces any previously registered + * callback for the same @p type. */ void addReadOnlyRefCreationFnc(const std::string& type, DataStoreRefCreateFnc creationFnc); /** - * @brief Adds a read-only reference list store creation function for the specified type. - * @param type The format name - * @param creationFnc The creation function to add + * @brief Registers a factory callback that creates read-only reference + * ListStores (NeighborList) for the given format name. + * + * Derived IO managers call this during construction to advertise their + * ability to create OOC ListStores. DataIOCollection::createReadOnlyRefListStore() + * dispatches to the callback registered here. + * + * @param type The format name to register under (e.g., "HDF5-OOC") + * @param creationFnc The factory callback. Replaces any previously registered + * callback for the same @p type. */ void addListStoreRefCreationFnc(const std::string& type, ListStoreRefCreateFnc creationFnc); /** - * @brief Adds a string store creation function for the specified type. - * @param type The format name - * @param creationFnc The creation function to add + * @brief Registers a factory callback that creates StringStores for the given + * format name. + * + * Derived IO managers call this during construction to advertise their + * ability to create StringStores (in-memory or OOC). DataIOCollection::createStringStore() + * dispatches to the callback registered here. + * + * @param type The format name to register under + * @param creationFnc The factory callback. Replaces any previously registered + * callback for the same @p type. */ void addStringStoreCreationFnc(const std::string& type, StringStoreCreateFnc creationFnc); /** - * @brief Adds a finalize-stores callback for the specified type. - * Called after pipeline execution to prepare stores for reading (e.g., close - * write handles and switch to read-only mode). - * @param type The data store type name - * @param fnc The callback to add + * @brief Registers a post-pipeline finalization callback for the given format. + * + * The callback is invoked by DataIOCollection::finalizeStores() after pipeline + * execution completes. It should walk the DataStructure and transition any + * stores owned by this format from write mode to read-only mode (e.g., closing + * HDF5 write handles and re-opening as read handles). + * + * @param type The format name to register under + * @param fnc The finalization callback. Replaces any previously registered + * callback for the same @p type. */ void addFinalizeStoresFnc(const std::string& type, FinalizeStoresFnc fnc); @@ -233,9 +351,9 @@ class SIMPLNX_EXPORT IDataIOManager factory_collection m_FactoryCollection; DataStoreCreationMap m_DataStoreCreationMap; ListStoreCreationMap m_ListStoreCreationMap; - DataStoreRefCreationMap m_ReadOnlyRefCreationMap; - ListStoreRefCreationMap m_ListStoreRefCreationMap; - StringStoreCreationMap m_StringStoreCreationMap; - FinalizeStoresMap m_FinalizeStoresMap; + DataStoreRefCreationMap m_ReadOnlyRefCreationMap; ///< OOC read-only DataStore factories keyed by format name + ListStoreRefCreationMap m_ListStoreRefCreationMap; ///< OOC read-only ListStore factories keyed by format name + StringStoreCreationMap m_StringStoreCreationMap; ///< StringStore factories keyed by format name + FinalizeStoresMap m_FinalizeStoresMap; ///< Post-pipeline finalize callbacks keyed by format name }; } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp index d8ff27a5d1..ce3c31a52f 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp @@ -49,19 +49,37 @@ inline Result<> WriteDataStore(nx::core::HDF5::DatasetIO& datasetWriter, const A } /** - * @brief Reads a DataStore from an HDF5 dataset. + * @brief Reads a DataStore from an HDF5 dataset, selecting between three + * loading strategies based on recovery-file metadata and OOC preferences. * - * For arrays that exceed the user's largeDataSize threshold and have an OOC - * format configured (e.g. "HDF5-OOC"), this creates a read-only reference - * store pointing at the source file instead of copying data into a temp file. - * Small arrays and in-core formats load normally via CreateDataStore + readHdf5. + * The three paths, tried in order, are: * - * The OOC interception is transparent: if no OOC plugin is loaded (no - * DataStoreRefCreateFnc registered), createReadOnlyRefStore returns nullptr - * and the code falls through to the normal in-core path. + * 1. **Recovery-file reattach** -- If the dataset carries OocBackingFilePath / + * OocBackingDatasetPath / OocChunkShape string attributes, this is a + * placeholder dataset written by a recovery file. The method reattaches to + * the original backing file via createReadOnlyRefStore(). If the backing + * file is missing or the OOC plugin is not loaded, falls through. * + * 2. **OOC format resolution** -- For arrays whose byte size exceeds the + * user's largeDataSize threshold (or if force-OOC is enabled), the + * plugin-registered format resolver selects an OOC format string (e.g. + * "HDF5-OOC"). A read-only reference store is created that points at + * the *source* HDF5 file, avoiding a multi-gigabyte copy into a temp + * file. Chunk shapes are derived from the HDF5 dataset's chunk layout; + * for contiguous datasets a default shape is synthesized (one Z-slice + * for 3D data, or clamped-to-64 for 1D/2D data). + * + * 3. **Normal in-core load** -- Creates a standard in-memory DataStore via + * CreateDataStore and reads all data from HDF5 into RAM. + * + * Each OOC path is guarded: if no OOC plugin is loaded (no factory + * registered for the format), createReadOnlyRefStore() returns nullptr and + * the code falls through transparently to the next strategy. + * + * @tparam T The element type of the data store * @param datasetReader The HDF5 dataset to read from - * @return std::shared_ptr> The created data store + * @return std::shared_ptr> The created data store, + * either in-core or backed by an OOC reference store */ template inline std::shared_ptr> ReadDataStore(const nx::core::HDF5::DatasetIO& datasetReader) @@ -69,13 +87,23 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: auto tupleShape = IDataStoreIO::ReadTupleShape(datasetReader); auto componentShape = IDataStoreIO::ReadComponentShape(datasetReader); - // Recovery file support: detect OOC placeholder datasets that have - // backing file metadata attributes. Recovery files store OOC arrays - // as scalar placeholders with OocBackingFilePath/OocBackingDatasetPath/ - // OocChunkShape attributes. If present, reattach from the backing file - // instead of reading the placeholder data. - // No compile-time guard — createReadOnlyRefStore returns nullptr if no - // OOC plugin is loaded, providing a graceful fallback. + // ----------------------------------------------------------------------- + // PATH 1: Recovery-file reattach + // ----------------------------------------------------------------------- + // Recovery files (written by WriteRecoveryFile) store OOC arrays as + // zero-byte scalar placeholder datasets annotated with three string + // attributes: + // - OocBackingFilePath : absolute path to the original backing HDF5 file + // - OocBackingDatasetPath: HDF5 dataset path inside the backing file + // - OocChunkShape : comma-separated chunk dimensions (e.g. "1,200,200") + // + // If these attributes exist and the backing file is still on disk, we + // create a read-only reference store pointing at the backing file instead + // of reading the (empty) placeholder data. This allows recovery files to + // rehydrate OOC arrays without duplicating the data. + // + // No compile-time guard is needed: createReadOnlyRefStore returns nullptr + // if no OOC plugin is loaded, and we fall through to path 2/3. { auto backingPathResult = datasetReader.readStringAttribute("OocBackingFilePath"); if(backingPathResult.valid() && !backingPathResult.value().empty()) @@ -84,10 +112,13 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: if(std::filesystem::exists(backingFilePath)) { + // Read the HDF5-internal dataset path from the backing file attribute. auto datasetPathResult = datasetReader.readStringAttribute("OocBackingDatasetPath"); std::string backingDatasetPath = datasetPathResult.valid() ? datasetPathResult.value() : ""; - // Parse chunk shape from comma-separated string (e.g. "1,200,200") + // Parse the chunk shape from a comma-separated string attribute + // (e.g. "1,200,200" for one Z-slice of a 3D dataset). The chunk + // shape tells the OOC store how to partition reads/writes. auto chunkResult = datasetReader.readStringAttribute("OocChunkShape"); ShapeType chunkShape; if(chunkResult.valid() && !chunkResult.value().empty()) @@ -103,13 +134,19 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } } + // Ask the IO collection to create a read-only reference store backed + // by the original file. The format is always "HDF5-OOC" for recovery + // files since that is what wrote the placeholder. auto ioCollection = DataStoreUtilities::GetIOCollection(); auto refStore = ioCollection->createReadOnlyRefStore("HDF5-OOC", GetDataType(), backingFilePath, backingDatasetPath, tupleShape, componentShape, chunkShape); if(refStore != nullptr) { + // Successfully reattached; wrap the IDataStore unique_ptr in a + // shared_ptr> via dynamic_pointer_cast. return std::shared_ptr>(std::dynamic_pointer_cast>(std::shared_ptr(std::move(refStore)))); } - // If createReadOnlyRefStore returns nullptr, fall through to normal path + // createReadOnlyRefStore returned nullptr (OOC plugin not loaded); + // fall through to path 2/3. } else { @@ -119,23 +156,27 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } } - // Determine what format this array should use based on its byte size - // and the user's OOC preferences (largeDataSize threshold, force flag). - // The plugin-registered resolver encapsulates all format-selection policy. + // ----------------------------------------------------------------------- + // PATH 2: OOC format resolution for large arrays + // ----------------------------------------------------------------------- + // Ask the plugin-registered format resolver whether this array should be + // stored out-of-core. The resolver checks the array's total byte size + // against the user's largeDataSize threshold and the force-OOC flag. + // If the resolver returns an OOC format (e.g. "HDF5-OOC"), we create a + // read-only reference store pointing directly at the *source* HDF5 file + // on disk, avoiding a multi-gigabyte copy into a temporary file. uint64 dataSize = DataStoreUtilities::CalculateDataSize(tupleShape, componentShape); auto ioCollection = DataStoreUtilities::GetIOCollection(); std::string dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, componentShape, dataSize); - // If the format is an OOC format (not empty, not explicit in-core), and - // we're reading from an existing HDF5 file, create a read-only reference - // store pointing at the source file. This avoids copying gigabytes of - // data into a temporary file for arrays that already exist on disk. if(!dataFormat.empty() && dataFormat != Preferences::k_InMemoryFormat) { + // Resolve the HDF5 file path for the source dataset. getFilePath() + // may return empty when the DatasetIO was created via a parentId + // constructor (GroupIO::openDataset); in that case, fall back to the + // HDF5 C API (H5Fget_name) to retrieve the file name from the + // dataset's HDF5 object ID. auto filePath = datasetReader.getFilePath(); - // ObjectIO::getFilePath() may return empty when the DatasetIO was - // created via the parentId constructor (GroupIO::openDataset). - // Fall back to querying HDF5 directly for the file name. if(filePath.empty()) { hid_t objId = datasetReader.getId(); @@ -150,21 +191,30 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } } } + // The HDF5-internal dataset path (e.g. "/DataStructure/Image/CellData/FeatureIds"). auto datasetPath = datasetReader.getObjectPath(); + + // Determine chunk shape in *tuple space* for the OOC reference store. + // HDF5 chunk dimensions include both tuple and component dimensions + // (the component dimensions are the trailing axes); we strip those and + // keep only the leading tuple-rank dimensions. + // + // For contiguous (non-chunked) datasets, getChunkDimensions() returns + // an empty vector, so we synthesize a default chunk shape: + // - 3D+ data: {1, Y, X, ...} -- one Z-slice per logical chunk, + // optimized for the common case of Z-slice-sequential access. + // - 1D/2D data: each dimension clamped to 64 elements. auto chunkDims = datasetReader.getChunkDimensions(); - // Compute chunk shape in tuple space. HDF5 chunk dims include both - // tuple and component dimensions; we take only the tuple portion. - // For contiguous datasets (empty chunkDims), use a default shape - // optimized for Z-slice-sequential access. ShapeType chunkShape; if(!chunkDims.empty() && chunkDims.size() >= tupleShape.size()) { + // Chunked dataset: extract the tuple-rank leading dimensions. chunkShape.assign(chunkDims.begin(), chunkDims.begin() + static_cast(tupleShape.size())); } else if(tupleShape.size() >= 3) { - // 3D+ contiguous: {1, Y, X, ...} — one Z-slice per logical chunk + // 3D+ contiguous dataset: default to one Z-slice per chunk {1, Y, X, ...}. chunkShape.resize(tupleShape.size()); chunkShape[0] = 1; for(usize d = 1; d < tupleShape.size(); ++d) @@ -174,7 +224,7 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } else { - // 1D/2D contiguous: clamp each dimension to 64 + // 1D/2D contiguous dataset: clamp each axis to 64 elements per chunk. chunkShape.resize(tupleShape.size()); for(usize d = 0; d < tupleShape.size(); ++d) { @@ -182,9 +232,10 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } } - // createReadOnlyRefStore returns nullptr if no ref factory is registered - // (e.g. SimplnxOoc plugin not loaded). In that case, fall through to - // the normal in-core path below. + // Attempt to create a read-only reference store backed by the source file. + // Returns nullptr if no factory is registered for this format (e.g. + // SimplnxOoc plugin not loaded); in that case, fall through to the + // normal in-core path below. auto refStore = ioCollection->createReadOnlyRefStore(dataFormat, GetDataType(), filePath, datasetPath, tupleShape, componentShape, chunkShape); if(refStore != nullptr) { @@ -192,7 +243,11 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } } - // Normal path: create an in-core DataStore and read the data from HDF5. + // ----------------------------------------------------------------------- + // PATH 3: Normal in-core load + // ----------------------------------------------------------------------- + // No OOC format was selected (or no OOC plugin is loaded). Allocate a + // standard in-memory DataStore and read the full dataset from HDF5. auto dataStore = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); dataStore->readHdf5(datasetReader); return dataStore; diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp index 462fdd6bf9..3df7df23df 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp @@ -118,9 +118,19 @@ Result<> DataStructureWriter::writeDataObject(const DataObject* dataObject, nx:: return writeDataObjectLink(dataObject, parentGroup); } - // Write-array-override hook: if active, give the registered plugin a chance - // to handle this object (e.g., writing a recovery placeholder for OOC arrays). - // If the hook returns std::nullopt, fall through to the normal write path. + // ----------------------------------------------------------------------- + // Write-array-override hook (OOC recovery-file support) + // ----------------------------------------------------------------------- + // When a WriteArrayOverrideGuard is on the stack (e.g. during + // WriteRecoveryFile), a plugin-registered callback intercepts DataArray + // writes. For OOC-backed arrays the callback writes a zero-byte + // placeholder dataset annotated with OocBackingFilePath / + // OocBackingDatasetPath / OocChunkShape attributes, so the recovery + // file stays small while preserving enough metadata to reattach to the + // backing file on reload (see ReadDataStore path 1). + // + // If the callback returns std::nullopt the object is not OOC-backed, so + // we fall through to the normal HDF5 write path below. auto ioCollection = DataStoreUtilities::GetIOCollection(); if(ioCollection->isWriteArrayOverrideActive()) { @@ -129,6 +139,7 @@ Result<> DataStructureWriter::writeDataObject(const DataObject* dataObject, nx:: { return overrideResult.value(); } + // std::nullopt -- not an OOC-backed object; continue to normal write. } // Normal write path diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp index 4403359b0a..a6da75d8cc 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp @@ -75,12 +75,24 @@ class SIMPLNX_EXPORT DataStructureWriter /** * @brief Writes the DataObject under the given GroupIO. If the - * DataObject has already been written, a link is create instead. + * DataObject has already been written, a link is created instead. + * + * Before using the normal type-factory write path, this method checks + * two conditions in order: + * + * 1. **Deduplication** -- If the DataObject has already been written to + * this file, an HDF5 hard link is created instead of a duplicate copy. + * + * 2. **Write-array-override hook** -- If a WriteArrayOverrideGuard is + * active (e.g. during WriteRecoveryFile), the registered plugin + * callback gets a chance to write OOC-backed arrays as lightweight + * placeholder datasets. If the callback returns std::nullopt the + * object is not OOC-backed, and the normal path is used. * * If the process encounters an error, the error code is returned. Otherwise, * this method returns 0. - * @param dataObject - * @param parentGroup + * @param dataObject The DataObject to write + * @param parentGroup The HDF5 group to write the object into * @return Result<> */ Result<> writeDataObject(const DataObject* dataObject, GroupIO& parentGroup); diff --git a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp index d306a4b806..98b58615b7 100644 --- a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp @@ -33,11 +33,20 @@ class NeighborListIO : public IDataIO ~NeighborListIO() noexcept override = default; /** - * @brief Attempts to read the NeighborList data from HDF5. - * Returns a Result<> with any errors or warnings encountered during the process. - * @param parentGroup - * @param dataReader - * @return Result<> + * @brief Reads NeighborList data from an HDF5 dataset. + * + * When useEmptyDataStore is true, only the TupleDimensions attribute from + * the linked NumNeighbors dataset is read, and an EmptyListStore placeholder + * is returned. The actual data is loaded later by finishImportingData(). + * + * When useEmptyDataStore is false, the full flat data array is read from + * HDF5, split into per-tuple vectors using the NumNeighbors companion + * array, and packed into an in-memory ListStore. + * + * @param parentGroup The HDF5 group containing the dataset and its companion + * @param dataReader The HDF5 dataset containing the flat packed neighbor data + * @param useEmptyDataStore If true, return an EmptyListStore placeholder + * @return std::shared_ptr The created list store, or nullptr on error */ static std::shared_ptr ReadHdf5Data(const nx::core::HDF5::GroupIO& parentGroup, const nx::core::HDF5::DatasetIO& dataReader, bool useEmptyDataStore = false) { @@ -128,10 +137,24 @@ class NeighborListIO : public IDataIO } /** - * @brief Replaces the AbstractListStore using data from the HDF5 dataset. - * @param dataStructure - * @param dataPath - * @param dataStructureReader + * @brief Replaces the placeholder AbstractListStore with real data from the + * HDF5 dataset. This is the "backfill" step called after preflight when the + * DataStructure was initially loaded with empty stores. + * + * The method follows the same two-path strategy as DataStoreIO::ReadDataStore: + * + * 1. **OOC path** -- If the format resolver selects an OOC format and a + * read-only reference list store factory is registered, create an OOC + * store backed by the flat neighbor data in the source HDF5 file. The + * chunk tuple count is one Z-slice for 3D+ tuple shapes, or the full + * tuple count for 1D/2D data. + * + * 2. **In-memory path** -- Read the flat data array from HDF5 and scatter + * it into per-tuple vectors in an in-memory ListStore. + * + * @param dataStructure The DataStructure containing the NeighborList to populate + * @param dataPath Path to the NeighborList in the DataStructure + * @param parentGroup The HDF5 group containing the dataset * @return Result<> */ Result<> finishImportingData(DataStructure& dataStructure, const DataPath& dataPath, const group_reader_type& parentGroup) const override @@ -152,12 +175,15 @@ class NeighborListIO : public IDataIO } numNeighborsName = std::move(numNeighborsNameResult.value()); + // Read the "NumNeighbors" companion array, which stores the per-tuple + // neighbor count used to interpret the flat packed data array. auto numNeighborsReader = parentGroup.openDataset(numNeighborsName); auto numNeighborsPtr = DataStoreIO::ReadDataStore(numNeighborsReader); auto& numNeighborsStore = *numNeighborsPtr.get(); - // Compute total neighbors and tuple shape before deciding in-memory vs OOC. - // The total number of elements in the flat dataset is the sum of all NumNeighbors values. + // Compute the total number of neighbor entries across all tuples and + // the corresponding byte size. These are needed to ask the format + // resolver whether this list should be stored out-of-core. const auto numTuples = numNeighborsStore.getNumberOfTuples(); const auto tupleShape = numNeighborsStore.getTupleShape(); uint64 totalNeighbors = 0; @@ -166,19 +192,19 @@ class NeighborListIO : public IDataIO totalNeighbors += static_cast(numNeighborsStore[i]); } - // Query the format resolver for OOC-vs-in-memory decision. The resolver is - // plugin-registered (SimplnxOoc); if no plugin is loaded, the format is empty - // and we fall through to in-memory loading. Also check that a read-only list - // store factory is registered for the format; if not, fall through. + // Query the plugin-registered format resolver. If an OOC format is + // selected (e.g. "HDF5-OOC") *and* a read-only list store factory + // exists for that format, we create an OOC store. Otherwise we load + // the full flat array into memory and scatter it into per-tuple lists. auto ioCollection = DataStoreUtilities::GetIOCollection(); const uint64 dataSize = totalNeighbors * sizeof(T); const std::string dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, {1}, dataSize); if(!dataFormat.empty() && ioCollection->hasReadOnlyRefListCreationFnc(dataFormat)) { - // Resolve the backing file path. GroupIO::getFilePath() may return empty when - // the GroupIO was created via a parentId constructor; fall back to HDF5's - // H5Fget_name using the dataset's id. + // Resolve the backing HDF5 file path. GroupIO::getFilePath() may + // return empty when the GroupIO was created via a parentId + // constructor; fall back to H5Fget_name using the dataset's id. std::filesystem::path filePath = parentGroup.getFilePath(); if(filePath.empty()) { @@ -198,15 +224,21 @@ class NeighborListIO : public IDataIO // HDF5-internal dataset path for the flat neighbor data array. const std::string datasetPath = dataReader.getObjectPath(); - // Chunk tuple count: for 3D+ tuple shapes use a single Z-slice (Y*X), - // otherwise chunk over the total tuple count. + // Compute the chunk tuple count for the OOC list store. For 3D+ + // tuple shapes, chunk by one Z-slice (Y * X tuples) to match the + // common slab-sequential access pattern. For 1D/2D data, chunk + // over the entire tuple count (the list data is small enough to + // read in a single pass). const usize chunkTupleCount = (tupleShape.size() >= 3) ? (tupleShape[tupleShape.size() - 2] * tupleShape[tupleShape.size() - 1]) : std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>()); const ShapeType chunkShape = {chunkTupleCount}; + // Create the OOC list store backed by the source HDF5 file. auto ocStore = ioCollection->createReadOnlyRefListStore(dataFormat, GetDataType(), filePath, datasetPath, tupleShape, chunkShape); if(ocStore != nullptr) { + // The factory returns a type-erased IListStore; downcast to the + // concrete AbstractListStore and install it into the NeighborList. auto* rawStore = ocStore.release(); auto* typedStore = dynamic_cast*>(rawStore); if(typedStore != nullptr) @@ -215,12 +247,14 @@ class NeighborListIO : public IDataIO neighborList.setStore(sharedStore); return {}; } - // dynamic_cast failed, clean up and fall through to in-memory path + // dynamic_cast failed (unexpected type mismatch); clean up and + // fall through to the in-memory path. delete rawStore; } } - // In-memory path: read the flat data and scatter into per-tuple lists. + // In-memory path: read the entire flat data array from HDF5 and + // scatter it into per-tuple vectors in an in-memory ListStore. auto flatDataStorePtr = dataReader.template readAsDataStore(); if(flatDataStorePtr == nullptr) { diff --git a/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp b/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp index 9eeaf3feb9..98176e015e 100644 --- a/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp +++ b/src/simplnx/DataStructure/IO/HDF5/StringArrayIO.cpp @@ -56,9 +56,13 @@ Result<> StringArrayIO::readData(DataStructureReader& dataStructureReader, const StringArray* data = nullptr; if(useEmptyDataStore) { - // Import with an empty vector to avoid allocating numValues empty strings, then swap in - // an EmptyStringStore placeholder. The actual string data will be loaded later by the - // backfill handler. + // During preflight (useEmptyDataStore == true), we create the StringArray + // with an empty string vector to avoid allocating potentially millions of + // std::string objects that would never be used. We then immediately swap + // the underlying store for an EmptyStringStore placeholder that reports + // the correct tuple shape/count but holds no data. The actual string + // content will be loaded later by finishImportingData() when the + // pipeline transitions from preflight to execution. data = StringArray::Import(dataStructureReader.getDataStructure(), dataArrayName, tupleShape, importId, std::vector{}, parentId); if(data != nullptr) { diff --git a/src/simplnx/Utilities/AlgorithmDispatch.hpp b/src/simplnx/Utilities/AlgorithmDispatch.hpp index 7735f5bf16..275101d069 100644 --- a/src/simplnx/Utilities/AlgorithmDispatch.hpp +++ b/src/simplnx/Utilities/AlgorithmDispatch.hpp @@ -51,8 +51,12 @@ inline bool AnyOutOfCore(std::initializer_list arrays) * to always select the out-of-core algorithm, regardless of storage type. * * This is primarily used in unit tests to exercise the OOC algorithm path - * even when data is stored in-core. Use ForceOocAlgorithmGuard for RAII-safe - * toggling in tests. + * even when data is stored in-core. The flag is backed by a function-local + * static, so it persists for the lifetime of the process. + * + * @warning This flag is NOT thread-safe. It should only be set from the main + * test thread before any parallel work begins. Use ForceOocAlgorithmGuard + * for RAII-safe toggling in tests. * * @return Reference to the static force flag */ @@ -100,6 +104,19 @@ inline const std::array k_ForceOocTestValues = {0, 1}; * @brief RAII guard that sets ForceOocAlgorithm() on construction and * restores the previous value on destruction. * + * The guard captures the current value of ForceOocAlgorithm() when constructed, + * overrides it with the requested value, and restores the original value when + * the guard goes out of scope. This ensures the global flag is always cleaned + * up, even if the test throws an exception or fails early. + * + * Copy and move operations are deleted to prevent accidental double-restore + * of the original value, which would corrupt the global flag state. + * + * @warning Not thread-safe. The underlying flag is a bare static bool with + * no synchronization. In Catch2 tests this is safe because each + * TEST_CASE runs on the main thread, but do not use this guard + * from worker threads. + * * Usage in tests: * @code * bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); @@ -112,11 +129,13 @@ class ForceOocAlgorithmGuard ForceOocAlgorithmGuard(bool force) : m_Original(ForceOocAlgorithm()) { + // Override the global flag for the duration of this guard's lifetime ForceOocAlgorithm() = force; } ~ForceOocAlgorithmGuard() { + // Restore the original value so subsequent tests start with a clean state ForceOocAlgorithm() = m_Original; } @@ -135,10 +154,15 @@ class ForceOocAlgorithmGuard * * This is primarily used in unit tests to exercise the in-core algorithm path * even when data is stored out-of-core (e.g., loaded from HDF5 in an OOC build). - * Use ForceInCoreAlgorithmGuard for RAII-safe toggling in tests. + * The flag is backed by a function-local static, so it persists for the lifetime + * of the process. * - * ForceInCoreAlgorithm() takes precedence over ForceOocAlgorithm() and - * AnyOutOfCore() in DispatchAlgorithm. + * ForceInCoreAlgorithm() takes the highest precedence in DispatchAlgorithm: + * when set to true, neither AnyOutOfCore() nor ForceOocAlgorithm() can + * override it. This allows tests to verify in-core correctness even when + * running in an OOC-enabled build where arrays may be loaded as chunked stores. + * + * @warning Not thread-safe. See ForceOocAlgorithm() for details. * * @return Reference to the static force flag */ @@ -149,11 +173,17 @@ inline bool& ForceInCoreAlgorithm() } /** - * @brief RAII guard that sets ForceInCoreAlgorithm() on construction and - * restores the previous value on destruction. + * @brief RAII guard that unconditionally sets ForceInCoreAlgorithm() to true + * on construction and restores the previous value on destruction. + * + * Unlike ForceOocAlgorithmGuard, this guard always forces in-core mode and + * does not accept a boolean parameter. This is intentional: forcing in-core + * is an override that should only be applied deliberately in tests that need + * to verify in-core behavior in an OOC-enabled build. + * + * Copy and move operations are deleted to prevent accidental double-restore. * - * Use this in tests that verify the in-core algorithm path when data may be - * loaded from disk as OOC (e.g., dream3d files in an OOC-enabled build). + * @warning Not thread-safe. See ForceOocAlgorithmGuard for details. * * Usage in tests: * @code @@ -166,11 +196,13 @@ class ForceInCoreAlgorithmGuard ForceInCoreAlgorithmGuard() : m_Original(ForceInCoreAlgorithm()) { + // Unconditionally force in-core dispatch for the guard's lifetime ForceInCoreAlgorithm() = true; } ~ForceInCoreAlgorithmGuard() { + // Restore the original value so subsequent tests start with a clean state ForceInCoreAlgorithm() = m_Original; } @@ -210,12 +242,19 @@ class ForceInCoreAlgorithmGuard template Result<> DispatchAlgorithm(std::initializer_list arrays, ArgsT&&... args) { + // Selection priority (highest to lowest): + // 1. ForceInCoreAlgorithm == true -> InCoreAlgo (test override, wins over everything) + // 2. AnyOutOfCore(arrays) == true -> OocAlgo (real OOC data detected at runtime) + // 3. ForceOocAlgorithm == true -> OocAlgo (test override for exercising OOC path) + // 4. Default -> InCoreAlgo (all data is in-memory) if(!ForceInCoreAlgorithm() && (AnyOutOfCore(arrays) || ForceOocAlgorithm())) { + // Construct the OOC algorithm with the forwarded args and invoke operator()() return OocAlgo(std::forward(args)...)(); } else { + // Construct the in-core algorithm with the forwarded args and invoke operator()() return InCoreAlgo(std::forward(args)...)(); } } diff --git a/src/simplnx/Utilities/DataStoreUtilities.hpp b/src/simplnx/Utilities/DataStoreUtilities.hpp index f34b048f67..057c59e2f1 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.hpp +++ b/src/simplnx/Utilities/DataStoreUtilities.hpp @@ -27,12 +27,30 @@ uint64 CalculateDataSize(const ShapeType& tupleShape, const ShapeType& component } /** - * @brief Creates a DataStore with the given properties - * @tparam T Primitive Type (int, float, ...) - * @param tupleShape The Tuple Dimensions - * @param componentShape The component dimensions - * @param mode The mode to assume: PREFLIGHT or EXECUTE. Preflight will NOT allocate any storage. EXECUTE will allocate the memory/storage - * @return + * @brief Creates a DataStore with the given properties, choosing between in-memory + * and out-of-core storage based on the active format resolver. + * + * In Preflight mode, returns an EmptyDataStore that records shape metadata + * without allocating any storage. In Execute mode, the function consults the + * DataIOCollection's format resolver to determine the appropriate storage + * format (e.g., "HDF5-OOC" for out-of-core, or "" for in-memory). If the + * caller provides an explicit dataFormat, the resolver is bypassed. + * + * The format resolution flow in Execute mode is: + * 1. If dataFormat is non-empty, use it as-is (caller override). + * 2. Otherwise, call ioCollection->resolveFormat() which asks the registered + * plugin (e.g., SimplnxOoc) whether this array should be OOC based on + * its type, shape, and total byte size. + * 3. Pass the resolved format to createDataStoreWithType(), which creates + * either an in-memory DataStore (for "" or k_InMemoryFormat) or an + * OOC-backed store (for "HDF5-OOC" etc.). + * + * @tparam T Primitive type (int8, float32, uint64, etc.) + * @param tupleShape The tuple dimensions (e.g., {100, 200, 300} for a 3D volume) + * @param componentShape The component dimensions (e.g., {3} for a 3-component vector) + * @param mode PREFLIGHT returns an EmptyDataStore; EXECUTE allocates real storage + * @param dataFormat Optional explicit format name. If empty, the format resolver decides. + * @return Shared pointer to the created AbstractDataStore */ template std::shared_ptr> CreateDataStore(const ShapeType& tupleShape, const ShapeType& componentShape, IDataAction::Mode mode, std::string dataFormat = "") @@ -40,15 +58,23 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap switch(mode) { case IDataAction::Mode::Preflight: { + // Preflight: no storage allocated, just record the shape and format hint return std::make_unique>(tupleShape, componentShape, dataFormat); } case IDataAction::Mode::Execute: { + // Compute the total byte size so the format resolver can make size-based decisions uint64 dataSize = CalculateDataSize(tupleShape, componentShape); auto ioCollection = GetIOCollection(); + + // If no explicit format was requested, ask the registered resolver (if any) + // to determine whether this array should be in-memory or OOC if(dataFormat.empty()) { dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, componentShape, dataSize); } + + // Create the store using the resolved format. The IOCollection dispatches + // to the appropriate IDataIOManager based on the format string. return ioCollection->createDataStoreWithType(dataFormat, tupleShape, componentShape); } default: { @@ -57,15 +83,33 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap } } +/** + * @brief Creates a ListStore (backing store for NeighborList arrays) with the + * given properties, using the same format resolution as CreateDataStore. + * + * NeighborLists have variable-length per-tuple data, so the actual byte size + * is not known at creation time. The function estimates the size using a + * component count of 10 as a heuristic, which is sufficient for the format + * resolver to decide between in-memory and OOC storage. + * + * @tparam T Primitive type of the list elements + * @param tupleShape The tuple dimensions + * @param mode PREFLIGHT returns an EmptyListStore; EXECUTE allocates real storage + * @param dataFormat Optional explicit format name. If empty, the format resolver decides. + * @return Shared pointer to the created AbstractListStore + */ template std::shared_ptr> CreateListStore(const ShapeType& tupleShape, IDataAction::Mode mode = IDataAction::Mode::Execute, std::string dataFormat = "") { switch(mode) { case IDataAction::Mode::Preflight: { + // Preflight: no storage allocated, just record the tuple shape return std::make_unique>(tupleShape); } case IDataAction::Mode::Execute: { + // Estimate byte size with a heuristic component count of 10 for format resolution. + // The actual per-tuple list lengths are unknown until data is populated. uint64 dataSize = CalculateDataSize(tupleShape, {10}); auto ioCollection = GetIOCollection(); if(dataFormat.empty()) diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp index e25e07378b..b3dc17f743 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp @@ -1096,10 +1096,30 @@ Result readLegacyNodeConnectivityList(DataStructure& dataStructure return ConvertResultTo(std::move(voidResult), std::move(value)); } +/** + * @brief Creates a NeighborList from a legacy DREAM3D (SIMPL) HDF5 dataset. + * + * The preflight parameter is forwarded to ReadHdf5Data so that during preflight + * the function creates the NeighborList with an empty store (just shape metadata) + * rather than reading the full variable-length data from disk. This avoids + * materializing potentially large NeighborList data during pipeline validation, + * which would be wasteful and could exhaust memory in OOC workflows. + * + * @tparam T Element type of the NeighborList + * @param dataStructure Target DataStructure to insert the NeighborList into + * @param parentId Parent object ID for the new NeighborList + * @param parentReader HDF5 group reader for the parent AttributeMatrix + * @param datasetReader HDF5 dataset reader for the NeighborList dataset + * @param tupleDims Tuple dimensions read from the legacy "TupleDimensions" attribute + * @param preflight When true, creates an empty store without reading data + * @return Result<> indicating success or failure + */ template Result<> createLegacyNeighborList(DataStructure& dataStructure, DataObject ::IdType parentId, const nx::core::HDF5::GroupIO& parentReader, const nx::core::HDF5::DatasetIO& datasetReader, const ShapeType& tupleDims, bool preflight = false) { + // Read the NeighborList data from HDF5. In preflight mode, this returns + // an empty store with the correct tuple count but no actual list data. auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader, preflight); auto* neighborList = NeighborList::Create(dataStructure, datasetReader.getName(), listStore, parentId); if(neighborList == nullptr) @@ -2461,8 +2481,20 @@ Result<> DREAM3D::WriteFile(const std::filesystem::path& path, const DataStructu Result<> DREAM3D::WriteRecoveryFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline) { + // Obtain the global DataIOCollection so we can activate the write-array-override. + // The SimplnxOoc plugin registers a callback on this collection at startup that + // knows how to write OOC array placeholders instead of full data. auto ioCollection = DataStoreUtilities::GetIOCollection(); + + // The RAII guard sets the override active on construction. While active, + // HDF5::DataStructureWriter will check each DataArray against the override + // callback before writing. The guard deactivates the override on destruction, + // ensuring it does not leak into subsequent normal WriteFile calls. WriteArrayOverrideGuard guard(ioCollection); + + // Delegate to the standard WriteFile path. The only difference is that the + // override is now active, so OOC arrays get placeholder writes. XDMF output + // is disabled (false) for recovery files since they are transient. return WriteFile(path, dataStructure, pipeline, false); } diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp index e88bbd48b8..fcd5a5daf9 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp @@ -100,15 +100,30 @@ SIMPLNX_EXPORT Result<> WriteFile(nx::core::HDF5::FileIO& fileWriter, const Pipe SIMPLNX_EXPORT Result<> WriteFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline = {}, bool writeXdmf = false); /** - * @brief Writes a recovery .dream3d file. Activates the write-array-override - * hook so that plugins (e.g., SimplnxOoc) can write lightweight placeholders - * for OOC arrays instead of materializing full data. In-core arrays are - * written normally via the standard write path. If no override is registered, - * behaves identically to WriteFile. - * @param path Output file path + * @brief Writes a recovery .dream3d file with optimized handling of OOC arrays. + * + * A recovery file captures the current pipeline state so that execution can + * be resumed after a crash or interruption. For OOC arrays, materializing + * the full data into HDF5 would be extremely expensive (potentially hundreds + * of GB). Instead, this function activates the DataIOCollection's + * write-array-override hook via a WriteArrayOverrideGuard RAII object. + * + * When the override is active, the HDF5 DataStructureWriter checks each + * DataArray against the registered override callback (set by the SimplnxOoc + * plugin). For OOC-backed arrays, the callback writes a lightweight + * placeholder (just the store metadata: file path, chunk layout, shape) + * instead of the full data. For in-core arrays, the callback returns + * std::nullopt, causing the writer to fall through to the standard HDF5 + * write path. + * + * If no write-array-override callback is registered (i.e., the OOC plugin + * is not loaded), the guard is a no-op and the function behaves identically + * to WriteFile. + * + * @param path Output file path for the recovery .dream3d file * @param dataStructure The DataStructure to write - * @param pipeline The Pipeline to write - * @return Result<> with any errors + * @param pipeline The Pipeline to serialize alongside the data (default empty) + * @return Result<> with any errors from the write operation */ SIMPLNX_EXPORT Result<> WriteRecoveryFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline = {}); diff --git a/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp b/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp index fa0021ab96..49c09d265d 100644 --- a/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp +++ b/src/simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.cpp @@ -961,15 +961,31 @@ Result<> DatasetIO::writeSpan(const DimsType& dims, nonstd::span Result<> DatasetIO::createEmptyDataset(const DimsType& dims) { + // Resolve the HDF5 native type ID for the template parameter. hid_t dataType = HdfTypeForPrimitive(); if(dataType == -1) { return MakeErrorResult(-1020, "createEmptyDataset error: Unsupported data type."); } + // Convert the DimsType vector to HDF5's hsize_t vector and create a + // simple N-D dataspace matching the full array dimensions. std::vector hDims(dims.size()); std::transform(dims.begin(), dims.end(), hDims.begin(), [](DimsType::value_type x) { return static_cast(x); }); hid_t dataspaceId = H5Screate_simple(static_cast(hDims.size()), hDims.data(), nullptr); @@ -978,6 +994,8 @@ Result<> DatasetIO::createEmptyDataset(const DimsType& dims) return MakeErrorResult(-1021, "createEmptyDataset error: Unable to create dataspace."); } + // Create (or reopen) the dataset. The dataset is left empty; data will + // be written later via writeSpanHyperslab(). auto datasetId = createOrOpenDataset(dataspaceId); H5Sclose(dataspaceId); if(datasetId < 0) @@ -988,6 +1006,25 @@ Result<> DatasetIO::createEmptyDataset(const DimsType& dims) return {}; } +// ----------------------------------------------------------------------------- +// writeSpanHyperslab +// ----------------------------------------------------------------------------- +// Writes a contiguous buffer of values into a rectangular sub-region (hyperslab) +// of an existing HDF5 dataset. The dataset must already exist on disk, created +// either by createEmptyDataset() or writeSpan(). +// +// This is the second step of the two-step OOC write pattern. An OOC store +// iterates over its backing file chunk-by-chunk, reads each chunk into a +// temporary buffer, and calls this method to write that buffer into the +// corresponding region of the output dataset. The pattern avoids ever +// materializing the entire array in memory. +// +// The method works by: +// 1. Opening the dataset's file dataspace +// 2. Selecting a hyperslab defined by start[] and count[] +// 3. Creating a compact memory dataspace matching count[] +// 4. Writing from the caller's span into the selected hyperslab +// ----------------------------------------------------------------------------- template Result<> DatasetIO::writeSpanHyperslab(nonstd::span values, const std::vector& start, const std::vector& count) { @@ -996,12 +1033,14 @@ Result<> DatasetIO::writeSpanHyperslab(nonstd::span values, const std:: return MakeErrorResult(-506, fmt::format("Cannot open HDF5 data at {} / {}", getFilePath().string(), getNamePath())); } + // Resolve the HDF5 native type for T. hid_t dataType = HdfTypeForPrimitive(); if(dataType == -1) { return MakeErrorResult(-1010, "writeSpanHyperslab error: Unsupported data type."); } + // Open the existing dataset and retrieve its file-side dataspace. hid_t datasetId = open(); hid_t fileSpaceId = H5Dget_space(datasetId); if(fileSpaceId < 0) @@ -1009,7 +1048,10 @@ Result<> DatasetIO::writeSpanHyperslab(nonstd::span values, const std:: return MakeErrorResult(-1011, "writeSpanHyperslab error: Unable to open the dataspace."); } - // Select hyperslab in the file dataspace + // Select the hyperslab region [start, start+count) in the file dataspace. + // On macOS, hsize_t (unsigned long long) differs from uint64 (unsigned long), + // so we must copy into vectors of the correct type to avoid mismatched + // pointer casts. #if defined(__APPLE__) std::vector startVec(start.begin(), start.end()); std::vector countVec(count.begin(), count.end()); @@ -1022,7 +1064,8 @@ Result<> DatasetIO::writeSpanHyperslab(nonstd::span values, const std:: return MakeErrorResult(-1012, "writeSpanHyperslab error: Unable to select hyperslab."); } - // Create memory dataspace matching the count dimensions + // Create a memory-side dataspace that matches the hyperslab extent. The + // caller's span must contain exactly product(count) elements. std::vector memDims(count.begin(), count.end()); hid_t memSpaceId = H5Screate_simple(static_cast(memDims.size()), memDims.data(), nullptr); if(memSpaceId < 0) @@ -1031,6 +1074,7 @@ Result<> DatasetIO::writeSpanHyperslab(nonstd::span values, const std:: return MakeErrorResult(-1013, "writeSpanHyperslab error: Unable to create memory dataspace."); } + // Write from the in-memory buffer into the selected hyperslab on disk. herr_t error = H5Dwrite(datasetId, dataType, memSpaceId, fileSpaceId, H5P_DEFAULT, values.data()); H5Sclose(memSpaceId); diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp index 891e39c8d8..f609439399 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp @@ -457,16 +457,34 @@ inline void CompareMontage(const AbstractMontage& exemplar, const AbstractMontag } /** - * @brief Compares two IDataArrays using bulk copyIntoBuffer for OOC efficiency. + * @brief Compares two IDataArrays element-by-element using bulk copyIntoBuffer + * for OOC-safe, high-performance comparison. * - * Reads both arrays in 40K-element chunks via copyIntoBuffer instead of - * per-element operator[], avoiding per-voxel OOC store overhead. - * NaN == NaN is treated as equal for floating-point types. + * Why copyIntoBuffer instead of operator[]: + * When arrays are backed by an out-of-core (chunked) DataStore, each call + * to operator[] may trigger a chunk load from disk. Comparing millions of + * elements one at a time would cause catastrophic chunk thrashing. Instead, + * this function reads both arrays in 40,000-element chunks via copyIntoBuffer, + * which batches HDF5 I/O and keeps access sequential. This is also safe for + * in-memory stores, where copyIntoBuffer is a simple memcpy. * - * @tparam T The element type - * @param left First array (typically exemplar) - * @param right Second array (typically generated) - * @param start Element index to start comparison from (default 0) + * Floating-point comparison semantics: + * - NaN == NaN is treated as equal. Many filter outputs legitimately produce + * NaN values (e.g., division by zero in optional statistics), and both the + * exemplar and generated arrays should agree on which elements are NaN. + * - Values within UnitTest::EPSILON of each other are treated as equal, + * accommodating floating-point rounding differences across platforms. + * + * Error reporting: + * On the first mismatched element, the function records the index and both + * values, then breaks out of the comparison loop. The mismatch details are + * reported via Catch2's UNSCOPED_INFO before the final REQUIRE(!failed). + * + * @tparam T The element type (must match the actual DataStore element type) + * @param left First array (typically the exemplar / golden reference) + * @param right Second array (typically the generated / computed result) + * @param start Element index to start comparison from (default 0). Useful when + * the first N elements are known to differ (e.g., header/padding). */ template void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize start = 0) @@ -477,6 +495,9 @@ void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize st INFO(fmt::format("Input Data Array:'{}' Output DataArray: '{}' bad comparison", left.getName(), right.getName())); REQUIRE(totalSize == newDataStore.getSize()); + // Use 40K-element chunks to balance memory usage against I/O efficiency. + // Each chunk is ~160 KB for float32 or ~320 KB for float64, which fits + // comfortably in L2 cache and aligns well with typical HDF5 chunk sizes. constexpr usize k_ChunkSize = 40000; auto oldBuf = std::make_unique(k_ChunkSize); auto newBuf = std::make_unique(k_ChunkSize); @@ -486,12 +507,16 @@ void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize st T failOld = {}; T failNew = {}; + // Iterate through the arrays in fixed-size chunks, reading both arrays + // into local buffers for fast element-wise comparison for(usize offset = start; offset < totalSize && !failed; offset += k_ChunkSize) { + // Handle the last chunk which may be smaller than k_ChunkSize const usize count = std::min(k_ChunkSize, totalSize - offset); oldDataStore.copyIntoBuffer(offset, nonstd::span(oldBuf.get(), count)); newDataStore.copyIntoBuffer(offset, nonstd::span(newBuf.get(), count)); + // Compare each element in the current chunk for(usize i = 0; i < count; i++) { const T oldVal = oldBuf[i]; @@ -500,17 +525,20 @@ void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize st { if constexpr(std::is_floating_point_v) { - // NaN == NaN is treated as equal + // Special case: NaN == NaN is treated as equal because many filters + // produce NaN for undefined results, and both arrays should agree if(std::isnan(oldVal) && std::isnan(newVal)) { continue; } + // Allow small floating-point differences within EPSILON tolerance float32 diff = std::fabs(static_cast(oldVal - newVal)); if(diff <= EPSILON) { continue; } } + // Record the first failure for diagnostic output, then stop failed = true; failIndex = offset + i; failOld = oldVal; From 549da6c6e996d73cf944f48e1b976c3f29d82d38 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Thu, 9 Apr 2026 08:41:07 -0400 Subject: [PATCH 03/13] REFACTOR: Move format resolution from DataStore creation to array creation layer Move the format resolver call site from the low-level DataStoreUtilities:: CreateDataStore/CreateListStore functions up to the array creation layer (ArrayCreationUtilities::CreateArray and ImportH5ObjectPathsAction). This is a prerequisite for the upcoming data store import handler refactor. Key architectural changes: 1. FormatResolverFnc signature expanded to (DataStructure, DataPath, DataType, dataSizeBytes). The resolver can now walk parent objects to determine geometry type, enabling it to force in-core for unstructured/ poly geometry arrays without caller-side checks. 2. Format resolution removed from DataStoreUtilities::CreateDataStore and CreateListStore. These are now simple factories that take an already- resolved format string. Callers are responsible for calling the resolver. 3. CreateArrayAction no longer carries a dataFormat member or constructor parameter. The k_DefaultDataFormat constant is removed. Format is resolved at execute time inside ArrayCreationUtilities::CreateArray. 4. ImportH5ObjectPathsAction gains a format-resolver loop that iterates Empty-store DataArrays after preflight import, consulting the resolver to decide which arrays to eager-load (in-core) vs leave for the backfill handler (OOC). 5. DataStoreIO::ReadDataStore and NeighborListIO::finishImportingData lose their inline format-resolution and OOC reference-store creation code. Format decisions for imported data are now made at the action level, not during raw HDF5 I/O. 6. Geometry actions (CreateGeometry1D/2D/3DAction, CreateVertexGeometry, CreateRectGridGeometry) lose their createdDataFormat parameter. They now materialize OOC topology arrays into in-core stores when the source arrays have StoreType::OutOfCore, since unstructured/poly geometry topology must be in-core for the visualization layer. 7. CheckMemoryRequirement simplified to a pure RAM check. OOC fallback logic removed since the resolver handles format decisions upstream. All filter callers updated to drop the dataFormat argument from CreateArrayAction constructors. Python binding updated (data_format parameter renamed to fill_value). Test files updated for new resolveFormat signature. --- .../Filters/ComputeFZQuaternionsFilter.cpp | 4 +- .../Filters/AppendImageGeometryFilter.cpp | 2 +- .../Filters/ComputeFeatureNeighborsFilter.cpp | 2 +- .../Filters/ComputeKMeansFilter.cpp | 5 +- .../Filters/ComputeKMedoidsFilter.cpp | 5 +- .../Filters/ComputeSurfaceFeaturesFilter.cpp | 4 +- .../Filters/ComputeVectorColorsFilter.cpp | 2 +- .../Filters/CreateDataArrayAdvancedFilter.cpp | 3 +- .../Filters/CreateDataArrayFilter.cpp | 4 +- ...eateFeatureArrayFromElementArrayFilter.cpp | 4 +- .../src/SimplnxCore/Filters/DBSCANFilter.cpp | 5 +- .../MapPointCloudToRegularGridFilter.cpp | 2 +- .../Filters/ReadTextDataArrayFilter.cpp | 4 +- .../Filters/ScalarSegmentFeaturesFilter.cpp | 4 +- .../SimplnxCore/Filters/SilhouetteFilter.cpp | 2 +- .../SimplnxCore/test/DREAM3DFileTest.cpp | 5 +- .../SimplnxCore/wrapping/python/simplnxpy.cpp | 2 +- .../IO/Generic/DataIOCollection.cpp | 4 +- .../IO/Generic/DataIOCollection.hpp | 27 +++--- .../DataStructure/IO/HDF5/DataStoreIO.hpp | 96 +------------------ .../DataStructure/IO/HDF5/NeighborListIO.hpp | 94 ++---------------- .../Filter/Actions/CreateArrayAction.cpp | 15 +-- .../Filter/Actions/CreateArrayAction.hpp | 14 +-- .../Filter/Actions/CreateGeometry1DAction.hpp | 40 +++++--- .../Filter/Actions/CreateGeometry2DAction.hpp | 40 +++++--- .../Filter/Actions/CreateGeometry3DAction.hpp | 40 +++++--- .../Actions/CreateRectGridGeometryAction.cpp | 8 +- .../Actions/CreateRectGridGeometryAction.hpp | 5 +- .../Actions/CreateVertexGeometryAction.hpp | 34 ++++--- .../Actions/ImportH5ObjectPathsAction.cpp | 78 +++++++++++++++ .../Utilities/ArrayCreationUtilities.cpp | 32 +------ .../Utilities/ArrayCreationUtilities.hpp | 43 ++++++--- src/simplnx/Utilities/DataStoreUtilities.hpp | 66 ++++--------- test/DataIOCollectionHooksTest.cpp | 19 ++-- test/IOFormat.cpp | 9 +- 35 files changed, 324 insertions(+), 399 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp index a77f6e6b2f..54cc97f6a5 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp @@ -122,8 +122,8 @@ IFilter::PreflightResult ComputeFZQuaternionsFilter::preflightImpl(const DataStr nx::core::Result resultOutputActions; - auto createArrayAction = std::make_unique(nx::core::DataType::float32, quatArray.getDataStore()->getTupleShape(), quatArray.getDataStore()->getComponentShape(), - pFZQuatsArrayPathValue, CreateArrayAction::k_DefaultDataFormat, "0.0"); + auto createArrayAction = + std::make_unique(nx::core::DataType::float32, quatArray.getDataStore()->getTupleShape(), quatArray.getDataStore()->getComponentShape(), pFZQuatsArrayPathValue, "0.0"); resultOutputActions.value().appendAction(std::move(createArrayAction)); // Return both the resultOutputActions and the preflightUpdatedValues via std::move() diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp index a26106d072..7c9c80d3f3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp @@ -307,7 +307,7 @@ IFilter::PreflightResult AppendImageGeometryFilter::preflightImpl(const DataStru auto compShape = inputDataArray != nullptr ? inputDataArray->getComponentShape() : destDataArray->getComponentShape(); auto cellDataDims = pSaveAsNewGeometry ? newCellDataDims : destCellDataDims; auto cellArrayPath = pSaveAsNewGeometry ? newArrayPath : destArrayPath; - auto createArrayAction = std::make_unique(dataType, cellDataDims, compShape, cellArrayPath, "", pDefaultValue); + auto createArrayAction = std::make_unique(dataType, cellDataDims, compShape, cellArrayPath, pDefaultValue); resultOutputActions.value().appendAction(std::move(createArrayAction)); } if(arrayType == IArray::ArrayType::NeighborListArray) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp index bc22762949..183dc57d7f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp @@ -158,7 +158,7 @@ IFilter::PreflightResult ComputeFeatureNeighborsFilter::preflightImpl(const Data // Create the SurfaceFeatures Output Data Array in the Feature Attribute Matrix if(storeSurfaceFeatures) { - auto action = std::make_unique(DataType::boolean, tupleShape, cDims, surfaceFeaturesPath, CreateArrayAction::k_DefaultDataFormat, "false"); + auto action = std::make_unique(DataType::boolean, tupleShape, cDims, surfaceFeaturesPath, "false"); actions.appendAction(std::move(action)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp index 91932895f9..abab2c1296 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp @@ -137,8 +137,7 @@ IFilter::PreflightResult ComputeKMeansFilter::preflightImpl(const DataStructure& } { - auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), - CreateArrayAction::k_DefaultDataFormat, "0"); + auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "0"); resultOutputActions.value().appendAction(std::move(createAction)); } @@ -146,7 +145,7 @@ IFilter::PreflightResult ComputeKMeansFilter::preflightImpl(const DataStructure& { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp index 7051340eb9..0bc5b16754 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp @@ -132,8 +132,7 @@ IFilter::PreflightResult ComputeKMedoidsFilter::preflightImpl(const DataStructur } { - auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), - CreateArrayAction::k_DefaultDataFormat, "0"); + auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "0"); resultOutputActions.value().appendAction(std::move(createAction)); } @@ -141,7 +140,7 @@ IFilter::PreflightResult ComputeKMedoidsFilter::preflightImpl(const DataStructur { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp index 5a07499910..75067685f2 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp @@ -109,8 +109,8 @@ IFilter::PreflightResult ComputeSurfaceFeaturesFilter::preflightImpl(const DataS tupleDims = surfaceFeaturesParent->getShape(); } - auto createSurfaceFeaturesAction = std::make_unique( - DataType::uint8, tupleDims, std::vector{1}, pCellFeaturesAttributeMatrixPathValue.createChildPath(pSurfaceFeaturesArrayNameValue), CreateArrayAction::k_DefaultDataFormat, "0"); + auto createSurfaceFeaturesAction = + std::make_unique(DataType::uint8, tupleDims, std::vector{1}, pCellFeaturesAttributeMatrixPathValue.createChildPath(pSurfaceFeaturesArrayNameValue), "0"); resultOutputActions.value().appendAction(std::move(createSurfaceFeaturesAction)); return {std::move(resultOutputActions), std::move(preflightUpdatedValues)}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp index 38e6ae243e..939170d438 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp @@ -109,7 +109,7 @@ IFilter::PreflightResult ComputeVectorColorsFilter::preflightImpl(const DataStru if(!pUseGoodVoxelsValue) { - auto action = std::make_unique(DataType::boolean, vectorsTupShape, std::vector{1}, k_MaskArrayPath, CreateArrayAction::k_DefaultDataFormat, "true"); + auto action = std::make_unique(DataType::boolean, vectorsTupShape, std::vector{1}, k_MaskArrayPath, "true"); resultOutputActions.value().appendAction(std::move(action)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp index a2890660b8..7785b8f414 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp @@ -161,7 +161,6 @@ IFilter::PreflightResult CreateDataArrayAdvancedFilter::preflightImpl(const Data auto compDimsData = filterArgs.value(k_CompDims_Key); auto dataArrayPath = filterArgs.value(k_DataPath_Key); auto tableData = filterArgs.value(k_TupleDims_Key); - auto dataFormat = filterArgs.value(k_DataFormat_Key); auto initFillValue = filterArgs.value(k_InitValue_Key); auto initIncFillValue = filterArgs.value(k_StartingFillValue_Key); auto stepValue = filterArgs.value(k_StepValue_Key); @@ -222,7 +221,7 @@ IFilter::PreflightResult CreateDataArrayAdvancedFilter::preflightImpl(const Data usize numTuples = std::accumulate(tupleDims.begin(), tupleDims.end(), static_cast(1), std::multiplies<>()); auto arrayDataType = ConvertNumericTypeToDataType(numericType); - auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath, dataFormat); + auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath); resultOutputActions.value().appendAction(std::move(action)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp index aec0be53ad..ccd4217974 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp @@ -110,8 +110,6 @@ IFilter::PreflightResult CreateDataArrayFilter::preflightImpl(const DataStructur auto dataArrayPath = filterArgs.value(k_DataPath_Key); auto initValue = filterArgs.value(k_InitializationValue_Key); auto tableData = filterArgs.value(k_TupleDims_Key); - auto dataFormat = filterArgs.value(k_DataFormat_Key); - nx::core::Result resultOutputActions; if(initValue.empty()) @@ -157,7 +155,7 @@ IFilter::PreflightResult CreateDataArrayFilter::preflightImpl(const DataStructur } // Sanity check that init value can be converted safely to the final numeric type integrated into action - auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath, dataFormat, initValue); + auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath, initValue); resultOutputActions.value().appendAction(std::move(action)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp index 81e9b6d415..ba31a15ea4 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp @@ -104,8 +104,8 @@ IFilter::PreflightResult CreateFeatureArrayFromElementArrayFilter::preflightImpl { DataType dataType = selectedCellArray.getDataType(); - auto createArrayAction = std::make_unique(dataType, amTupleShape, selectedCellArrayStore.getComponentShape(), - pCellFeatureAttributeMatrixPathValue.createChildPath(pCreatedArrayNameValue), CreateArrayAction::k_DefaultDataFormat, "0"); + auto createArrayAction = + std::make_unique(dataType, amTupleShape, selectedCellArrayStore.getComponentShape(), pCellFeatureAttributeMatrixPathValue.createChildPath(pCreatedArrayNameValue), "0"); resultOutputActions.value().appendAction(std::move(createArrayAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp index 8615e53289..e49ffd81b8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp @@ -160,8 +160,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt } { - auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), - CreateArrayAction::k_DefaultDataFormat, "0"); + auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "0"); resultOutputActions.value().appendAction(std::move(createAction)); } @@ -169,7 +168,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp index bb57e5683c..f09bdf078e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp @@ -351,7 +351,7 @@ IFilter::PreflightResult MapPointCloudToRegularGridFilter::preflightImpl(const D { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, vertexData->getShape(), std::vector{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true"); + auto createAction = std::make_unique(DataType::boolean, vertexData->getShape(), std::vector{1}, tempPath, "true"); actions.appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp index 296236ab5c..77e36d9c96 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp @@ -99,8 +99,6 @@ IFilter::PreflightResult ReadTextDataArrayFilter::preflightImpl(const DataStruct auto useDims = filterArgs.value(k_AdvancedOptions_Key); auto tableData = filterArgs.value(k_NTuples_Key); - auto dataFormat = filterArgs.value(k_DataFormat_Key); - nx::core::Result resultOutputActions; ShapeType tupleDims = {}; @@ -139,7 +137,7 @@ IFilter::PreflightResult ReadTextDataArrayFilter::preflightImpl(const DataStruct } } - auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, std::vector{nComp}, arrayPath, dataFormat); + auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, std::vector{nComp}, arrayPath); resultOutputActions.value().appendAction(std::move(action)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp index 636ca22ccf..2b64ac60eb 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp @@ -137,11 +137,11 @@ IFilter::PreflightResult ScalarSegmentFeaturesFilter::preflightImpl(const DataSt } // Create the Cell Level FeatureIds array - auto createFeatureIdsAction = std::make_unique(DataType::int32, cellTupleDims, std::vector{1}, featureIdsPath, createdArrayFormat, "0"); + auto createFeatureIdsAction = std::make_unique(DataType::int32, cellTupleDims, std::vector{1}, featureIdsPath, "0"); // Create the Feature Attribute Matrix auto createFeatureGroupAction = std::make_unique(cellFeaturesPath, std::vector{1}); - auto createActiveAction = std::make_unique(DataType::uint8, std::vector{1}, std::vector{1}, activeArrayPath, createdArrayFormat, "1"); + auto createActiveAction = std::make_unique(DataType::uint8, std::vector{1}, std::vector{1}, activeArrayPath, "1"); nx::core::Result resultOutputActions; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp index 96a439ef57..b2753a2aff 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp @@ -120,7 +120,7 @@ IFilter::PreflightResult SilhouetteFilter::preflightImpl(const DataStructure& da { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, CreateArrayAction::k_DefaultDataFormat, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp b/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp index 80d01581bc..8091e36d58 100644 --- a/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp +++ b/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp @@ -144,9 +144,8 @@ DataStructure CreateTestDataStructure() ShapeType tupleShape = {10}; auto* attributeMatrix = AttributeMatrix::Create(dataStructure, DataNames::k_AttributeMatrixName, tupleShape, group1->getId()); - Result<> arrayCreationResults = - ArrayCreationUtilities::CreateArray(dataStructure, tupleShape, std::vector{1}, DataPath({DataNames::k_Group1Name, DataNames::k_AttributeMatrixName, DataNames::k_Array2Name}), - IDataAction::Mode::Execute, ArrayCreationUtilities::k_DefaultDataFormat, "1"); + Result<> arrayCreationResults = ArrayCreationUtilities::CreateArray( + dataStructure, tupleShape, std::vector{1}, DataPath({DataNames::k_Group1Name, DataNames::k_AttributeMatrixName, DataNames::k_Array2Name}), IDataAction::Mode::Execute, "1"); return dataStructure; } diff --git a/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp b/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp index 239252013b..bea05d76cd 100644 --- a/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp +++ b/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp @@ -1175,7 +1175,7 @@ PYBIND11_MODULE(simplnx, mod) auto createArrayAction = SIMPLNX_PY_BIND_CLASS_VARIADIC(mod, CreateArrayAction, IDataCreationAction); createArrayAction.def(py::init&, const std::vector&, const DataPath&, std::string>(), "type"_a, "t_dims"_a, "c_dims"_a, "path"_a, - "data_format"_a = std::string("")); + "fill_value"_a = std::string("")); auto createAttributeMatrixAction = SIMPLNX_PY_BIND_CLASS_VARIADIC(mod, CreateAttributeMatrixAction, IDataCreationAction); createAttributeMatrixAction.def(py::init(), "path"_a, "shape"_a); diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index e76b2e32f3..388d790e8b 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -222,7 +222,7 @@ bool DataIOCollection::hasFormatResolver() const return static_cast(m_FormatResolver); } -std::string DataIOCollection::resolveFormat(DataType numericType, const ShapeType& tupleShape, const ShapeType& componentShape, uint64 dataSizeBytes) const +std::string DataIOCollection::resolveFormat(const DataStructure& dataStructure, const DataPath& arrayPath, DataType numericType, uint64 dataSizeBytes) const { // If no resolver is installed, return "" to signal the default in-memory format. if(!m_FormatResolver) @@ -231,7 +231,7 @@ std::string DataIOCollection::resolveFormat(DataType numericType, const ShapeTyp } // Delegate to the plugin-provided resolver. It returns a format name like // "HDF5-OOC" for large arrays, or "" to keep the default in-memory store. - return m_FormatResolver(numericType, tupleShape, componentShape, dataSizeBytes); + return m_FormatResolver(dataStructure, arrayPath, numericType, dataSizeBytes); } // --------------------------------------------------------------------------- diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index 00238d8eb7..42933fd5bc 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -47,8 +47,9 @@ class SIMPLNX_EXPORT DataIOCollection * @brief Callback that decides which storage format to use when a new array is * created during filter execution (e.g., in CreateArrayAction). * - * The resolver inspects the array's metadata (numeric type, shape, total byte - * size) and returns either: + * The resolver receives the full DataStructure and array DataPath so it can + * walk parent objects to determine geometry type, plus the numeric type and + * total byte size for size-threshold decisions. It returns either: * - A format name string (e.g., "HDF5-OOC") to request out-of-core storage, or * - An empty string "" to use the default in-memory DataStore. * @@ -56,13 +57,14 @@ class SIMPLNX_EXPORT DataIOCollection * storage without the core library needing to know about any specific OOC * format. Only one resolver may be active at a time. * - * @param numericType The element data type (float32, int32, uint8, etc.) - * @param tupleShape The tuple dimensions of the array - * @param componentShape The component dimensions per tuple - * @param dataSizeBytes The total size of the array data in bytes + * @param dataStructure The DataStructure containing (or about to contain) the array. + * The resolver can walk parent objects to determine geometry type. + * @param arrayPath The DataPath where the array lives or will be created. + * @param numericType The element data type (float32, int32, uint8, etc.) + * @param dataSizeBytes The total size of the array data in bytes * @return Format name string, or empty string for in-memory default */ - using FormatResolverFnc = std::function; + using FormatResolverFnc = std::function; /** * @brief Callback invoked once after a .dream3d file has been imported with @@ -145,14 +147,15 @@ class SIMPLNX_EXPORT DataIOCollection * resolver is registered, or the resolver returns "", the caller falls back * to the default in-memory DataStore. * - * @param numericType The element data type of the array - * @param tupleShape The tuple dimensions - * @param componentShape The component dimensions - * @param dataSizeBytes The total size of the array data in bytes + * @param dataStructure The DataStructure containing (or about to contain) the array. + * The resolver can walk parent objects to determine geometry type. + * @param arrayPath The DataPath where the array lives or will be created. + * @param numericType The element data type of the array + * @param dataSizeBytes The total size of the array data in bytes * @return A format name string (e.g., "HDF5-OOC") to use a specific format, * or an empty string for the default in-memory format */ - std::string resolveFormat(DataType numericType, const ShapeType& tupleShape, const ShapeType& componentShape, uint64 dataSizeBytes) const; + std::string resolveFormat(const DataStructure& dataStructure, const DataPath& arrayPath, DataType numericType, uint64 dataSizeBytes) const; /** * @brief Registers or clears the backfill handler callback. diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp index ce3c31a52f..5135bdf74d 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp @@ -1,6 +1,5 @@ #pragma once -#include "simplnx/Core/Preferences.hpp" #include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/IO/HDF5/IDataStoreIO.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" @@ -157,97 +156,12 @@ inline std::shared_ptr> ReadDataStore(const nx::core::HDF5: } // ----------------------------------------------------------------------- - // PATH 2: OOC format resolution for large arrays + // PATH 2: Normal in-core load // ----------------------------------------------------------------------- - // Ask the plugin-registered format resolver whether this array should be - // stored out-of-core. The resolver checks the array's total byte size - // against the user's largeDataSize threshold and the force-OOC flag. - // If the resolver returns an OOC format (e.g. "HDF5-OOC"), we create a - // read-only reference store pointing directly at the *source* HDF5 file - // on disk, avoiding a multi-gigabyte copy into a temporary file. - uint64 dataSize = DataStoreUtilities::CalculateDataSize(tupleShape, componentShape); - auto ioCollection = DataStoreUtilities::GetIOCollection(); - std::string dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, componentShape, dataSize); - - if(!dataFormat.empty() && dataFormat != Preferences::k_InMemoryFormat) - { - // Resolve the HDF5 file path for the source dataset. getFilePath() - // may return empty when the DatasetIO was created via a parentId - // constructor (GroupIO::openDataset); in that case, fall back to the - // HDF5 C API (H5Fget_name) to retrieve the file name from the - // dataset's HDF5 object ID. - auto filePath = datasetReader.getFilePath(); - if(filePath.empty()) - { - hid_t objId = datasetReader.getId(); - if(objId > 0) - { - ssize_t nameLen = H5Fget_name(objId, nullptr, 0); - if(nameLen > 0) - { - std::string nameBuf(static_cast(nameLen), '\0'); - H5Fget_name(objId, nameBuf.data(), static_cast(nameLen) + 1); - filePath = nameBuf; - } - } - } - // The HDF5-internal dataset path (e.g. "/DataStructure/Image/CellData/FeatureIds"). - auto datasetPath = datasetReader.getObjectPath(); - - // Determine chunk shape in *tuple space* for the OOC reference store. - // HDF5 chunk dimensions include both tuple and component dimensions - // (the component dimensions are the trailing axes); we strip those and - // keep only the leading tuple-rank dimensions. - // - // For contiguous (non-chunked) datasets, getChunkDimensions() returns - // an empty vector, so we synthesize a default chunk shape: - // - 3D+ data: {1, Y, X, ...} -- one Z-slice per logical chunk, - // optimized for the common case of Z-slice-sequential access. - // - 1D/2D data: each dimension clamped to 64 elements. - auto chunkDims = datasetReader.getChunkDimensions(); - - ShapeType chunkShape; - if(!chunkDims.empty() && chunkDims.size() >= tupleShape.size()) - { - // Chunked dataset: extract the tuple-rank leading dimensions. - chunkShape.assign(chunkDims.begin(), chunkDims.begin() + static_cast(tupleShape.size())); - } - else if(tupleShape.size() >= 3) - { - // 3D+ contiguous dataset: default to one Z-slice per chunk {1, Y, X, ...}. - chunkShape.resize(tupleShape.size()); - chunkShape[0] = 1; - for(usize d = 1; d < tupleShape.size(); ++d) - { - chunkShape[d] = tupleShape[d]; - } - } - else - { - // 1D/2D contiguous dataset: clamp each axis to 64 elements per chunk. - chunkShape.resize(tupleShape.size()); - for(usize d = 0; d < tupleShape.size(); ++d) - { - chunkShape[d] = std::min(static_cast(64), tupleShape[d]); - } - } - - // Attempt to create a read-only reference store backed by the source file. - // Returns nullptr if no factory is registered for this format (e.g. - // SimplnxOoc plugin not loaded); in that case, fall through to the - // normal in-core path below. - auto refStore = ioCollection->createReadOnlyRefStore(dataFormat, GetDataType(), filePath, datasetPath, tupleShape, componentShape, chunkShape); - if(refStore != nullptr) - { - return std::shared_ptr>(std::dynamic_pointer_cast>(std::shared_ptr(std::move(refStore)))); - } - } - - // ----------------------------------------------------------------------- - // PATH 3: Normal in-core load - // ----------------------------------------------------------------------- - // No OOC format was selected (or no OOC plugin is loaded). Allocate a - // standard in-memory DataStore and read the full dataset from HDF5. + // Format resolution for imported data is handled by the backfill strategy + // at a higher level (CreateArray / ImportH5ObjectPathsAction). During the + // eager HDF5 read path, we always load in-core. OOC decisions for imported + // data are made when the DataStructure is populated, not during raw I/O. auto dataStore = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); dataStore->readHdf5(datasetReader); return dataStore; diff --git a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp index 98b58615b7..2c7accab4b 100644 --- a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp @@ -11,10 +11,6 @@ #include "simplnx/DataStructure/NeighborList.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" -#include - -#include -#include #include namespace nx::core @@ -141,16 +137,9 @@ class NeighborListIO : public IDataIO * HDF5 dataset. This is the "backfill" step called after preflight when the * DataStructure was initially loaded with empty stores. * - * The method follows the same two-path strategy as DataStoreIO::ReadDataStore: - * - * 1. **OOC path** -- If the format resolver selects an OOC format and a - * read-only reference list store factory is registered, create an OOC - * store backed by the flat neighbor data in the source HDF5 file. The - * chunk tuple count is one Z-slice for 3D+ tuple shapes, or the full - * tuple count for 1D/2D data. - * - * 2. **In-memory path** -- Read the flat data array from HDF5 and scatter - * it into per-tuple vectors in an in-memory ListStore. + * Reads the flat data array from HDF5 and scatters it into per-tuple vectors + * in an in-memory ListStore. OOC format decisions for imported data are + * handled at a higher level by the backfill strategy. * * @param dataStructure The DataStructure containing the NeighborList to populate * @param dataPath Path to the NeighborList in the DataStructure @@ -181,80 +170,15 @@ class NeighborListIO : public IDataIO auto numNeighborsPtr = DataStoreIO::ReadDataStore(numNeighborsReader); auto& numNeighborsStore = *numNeighborsPtr.get(); - // Compute the total number of neighbor entries across all tuples and - // the corresponding byte size. These are needed to ask the format - // resolver whether this list should be stored out-of-core. const auto numTuples = numNeighborsStore.getNumberOfTuples(); const auto tupleShape = numNeighborsStore.getTupleShape(); - uint64 totalNeighbors = 0; - for(usize i = 0; i < numTuples; i++) - { - totalNeighbors += static_cast(numNeighborsStore[i]); - } - - // Query the plugin-registered format resolver. If an OOC format is - // selected (e.g. "HDF5-OOC") *and* a read-only list store factory - // exists for that format, we create an OOC store. Otherwise we load - // the full flat array into memory and scatter it into per-tuple lists. - auto ioCollection = DataStoreUtilities::GetIOCollection(); - const uint64 dataSize = totalNeighbors * sizeof(T); - const std::string dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, {1}, dataSize); - - if(!dataFormat.empty() && ioCollection->hasReadOnlyRefListCreationFnc(dataFormat)) - { - // Resolve the backing HDF5 file path. GroupIO::getFilePath() may - // return empty when the GroupIO was created via a parentId - // constructor; fall back to H5Fget_name using the dataset's id. - std::filesystem::path filePath = parentGroup.getFilePath(); - if(filePath.empty()) - { - hid_t objId = dataReader.getId(); - if(objId > 0) - { - ssize_t nameLen = H5Fget_name(objId, nullptr, 0); - if(nameLen > 0) - { - std::string nameBuf(static_cast(nameLen), '\0'); - H5Fget_name(objId, nameBuf.data(), static_cast(nameLen) + 1); - filePath = nameBuf; - } - } - } - - // HDF5-internal dataset path for the flat neighbor data array. - const std::string datasetPath = dataReader.getObjectPath(); - - // Compute the chunk tuple count for the OOC list store. For 3D+ - // tuple shapes, chunk by one Z-slice (Y * X tuples) to match the - // common slab-sequential access pattern. For 1D/2D data, chunk - // over the entire tuple count (the list data is small enough to - // read in a single pass). - const usize chunkTupleCount = (tupleShape.size() >= 3) ? (tupleShape[tupleShape.size() - 2] * tupleShape[tupleShape.size() - 1]) : - std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>()); - const ShapeType chunkShape = {chunkTupleCount}; - - // Create the OOC list store backed by the source HDF5 file. - auto ocStore = ioCollection->createReadOnlyRefListStore(dataFormat, GetDataType(), filePath, datasetPath, tupleShape, chunkShape); - if(ocStore != nullptr) - { - // The factory returns a type-erased IListStore; downcast to the - // concrete AbstractListStore and install it into the NeighborList. - auto* rawStore = ocStore.release(); - auto* typedStore = dynamic_cast*>(rawStore); - if(typedStore != nullptr) - { - auto sharedStore = std::shared_ptr>(typedStore); - neighborList.setStore(sharedStore); - return {}; - } - // dynamic_cast failed (unexpected type mismatch); clean up and - // fall through to the in-memory path. - delete rawStore; - } - } - // In-memory path: read the entire flat data array from HDF5 and - // scatter it into per-tuple vectors in an in-memory ListStore. + // Format resolution for imported data is handled by the backfill strategy + // at a higher level (CreateNeighborListAction / ImportH5ObjectPathsAction). + // During the eager HDF5 read path, we always load in-core. + // + // Read the entire flat data array from HDF5 and scatter it into + // per-tuple vectors in an in-memory ListStore. auto flatDataStorePtr = dataReader.template readAsDataStore(); if(flatDataStorePtr == nullptr) { diff --git a/src/simplnx/Filter/Actions/CreateArrayAction.cpp b/src/simplnx/Filter/Actions/CreateArrayAction.cpp index c5ea2caa40..1389d31e46 100644 --- a/src/simplnx/Filter/Actions/CreateArrayAction.cpp +++ b/src/simplnx/Filter/Actions/CreateArrayAction.cpp @@ -10,22 +10,20 @@ namespace struct CreateArrayFunctor { template - Result<> operator()(DataStructure& dataStructure, const std::vector& tDims, const std::vector& cDims, const DataPath& path, IDataAction::Mode mode, std::string dataFormat, - std::string fillValue) + Result<> operator()(DataStructure& dataStructure, const std::vector& tDims, const std::vector& cDims, const DataPath& path, IDataAction::Mode mode, std::string fillValue) { - return ArrayCreationUtilities::CreateArray(dataStructure, tDims, cDims, path, mode, dataFormat, fillValue); + return ArrayCreationUtilities::CreateArray(dataStructure, tDims, cDims, path, mode, fillValue); } }; } // namespace namespace nx::core { -CreateArrayAction::CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string dataFormat, std::string fillValue) +CreateArrayAction::CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string fillValue) : IDataCreationAction(path) , m_Type(type) , m_Dims(tDims) , m_CDims(cDims) -, m_DataFormat(dataFormat) , m_FillValue(fillValue) { } @@ -34,7 +32,7 @@ CreateArrayAction::~CreateArrayAction() noexcept = default; Result<> CreateArrayAction::apply(DataStructure& dataStructure, Mode mode) const { - return ExecuteDataFunction(::CreateArrayFunctor{}, m_Type, dataStructure, m_Dims, m_CDims, getCreatedPath(), mode, m_DataFormat, m_FillValue); + return ExecuteDataFunction(::CreateArrayFunctor{}, m_Type, dataStructure, m_Dims, m_CDims, getCreatedPath(), mode, m_FillValue); } IDataAction::UniquePointer CreateArrayAction::clone() const @@ -67,11 +65,6 @@ std::vector CreateArrayAction::getAllCreatedPaths() const return {getCreatedPath()}; } -std::string CreateArrayAction::dataFormat() const -{ - return m_DataFormat; -} - std::string CreateArrayAction::fillValue() const { return m_FillValue; diff --git a/src/simplnx/Filter/Actions/CreateArrayAction.hpp b/src/simplnx/Filter/Actions/CreateArrayAction.hpp index 8c43775aed..0d76909977 100644 --- a/src/simplnx/Filter/Actions/CreateArrayAction.hpp +++ b/src/simplnx/Filter/Actions/CreateArrayAction.hpp @@ -18,8 +18,6 @@ namespace nx::core class SIMPLNX_EXPORT CreateArrayAction : public IDataCreationAction { public: - inline static constexpr StringLiteral k_DefaultDataFormat = ""; - CreateArrayAction() = delete; /** @@ -28,10 +26,9 @@ class SIMPLNX_EXPORT CreateArrayAction : public IDataCreationAction * @param tDims The tuple dimensions * @param cDims The component dimensions * @param path The path where the DataArray will be created - * @param dataFormat The data format (empty string for in-memory) * @param fillValue The fill value for the array */ - CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string dataFormat = "", std::string fillValue = ""); + CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string fillValue = ""); ~CreateArrayAction() noexcept override; @@ -85,14 +82,6 @@ class SIMPLNX_EXPORT CreateArrayAction : public IDataCreationAction */ std::vector getAllCreatedPaths() const override; - /** - * @brief Returns the data formatting name for use in creating the appropriate data store. - * An empty string results in creating an in-memory DataStore. - * Other formats must be defined in external plugins. - * @return std::string - */ - std::string dataFormat() const; - /** * @brief Returns the fill value of the DataArray to be created. * @return std::string @@ -103,7 +92,6 @@ class SIMPLNX_EXPORT CreateArrayAction : public IDataCreationAction DataType m_Type; std::vector m_Dims; std::vector m_CDims; - std::string m_DataFormat = ""; std::string m_FillValue = ""; }; } // namespace nx::core diff --git a/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp b/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp index aaf4ca6b02..cca51870d8 100644 --- a/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp +++ b/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp @@ -3,8 +3,10 @@ #include "simplnx/Common/Array.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/Geometry/EdgeGeom.hpp" #include "simplnx/DataStructure/Geometry/IGeometry.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Filter/Output.hpp" #include "simplnx/Utilities/ArrayCreationUtilities.hpp" #include "simplnx/simplnx_export.hpp" @@ -35,7 +37,7 @@ class CreateGeometry1DAction : public IDataCreationAction * @param sharedEdgesName The name of the shared edge list array to be created */ CreateGeometry1DAction(const DataPath& geometryPath, size_t numEdges, size_t numVertices, const std::string& vertexAttributeMatrixName, const std::string& edgeAttributeMatrixName, - const std::string& sharedVerticesName, const std::string& sharedEdgesName, std::string createdDataFormat = "") + const std::string& sharedVerticesName, const std::string& sharedEdgesName) : IDataCreationAction(geometryPath) , m_NumEdges(numEdges) , m_NumVertices(numVertices) @@ -43,7 +45,6 @@ class CreateGeometry1DAction : public IDataCreationAction , m_EdgeDataName(edgeAttributeMatrixName) , m_SharedVerticesName(sharedVerticesName) , m_SharedEdgesName(sharedEdgesName) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -57,7 +58,7 @@ class CreateGeometry1DAction : public IDataCreationAction * @param arrayType Tells whether to copy, move, or reference the existing input vertices array */ CreateGeometry1DAction(const DataPath& geometryPath, const DataPath& inputVerticesArrayPath, const DataPath& inputEdgesArrayPath, const std::string& vertexAttributeMatrixName, - const std::string& edgeAttributeMatrixName, const ArrayHandlingType& arrayType, std::string createdDataFormat = "") + const std::string& edgeAttributeMatrixName, const ArrayHandlingType& arrayType) : IDataCreationAction(geometryPath) , m_VertexDataName(vertexAttributeMatrixName) , m_EdgeDataName(edgeAttributeMatrixName) @@ -66,7 +67,6 @@ class CreateGeometry1DAction : public IDataCreationAction , m_InputVertices(inputVerticesArrayPath) , m_InputEdges(inputEdgesArrayPath) , m_ArrayHandlingType(arrayType) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -140,11 +140,32 @@ class CreateGeometry1DAction : public IDataCreationAction DimensionType edgeTupleShape = {m_NumEdges}; DimensionType vertexTupleShape = {m_NumVertices}; // We probably don't know how many Vertices there are but take what ever the developer sends us - if(m_ArrayHandlingType == ArrayHandlingType::Copy) + // For Copy/Move/Reference, read shapes and materialize OOC stores upfront + if(m_ArrayHandlingType != ArrayHandlingType::Create) { edgeTupleShape = edges->getTupleShape(); vertexTupleShape = vertices->getTupleShape(); + // If the source arrays have OOC-backed stores, materialize them into + // in-core stores. These arrays may have been created OOC earlier in + // the pipeline when they lived outside any geometry. Unstructured/poly + // geometry topology arrays must be in-core for the visualization layer. + if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(vertexTupleShape, ShapeType{3}, std::optional{}); + vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + vertices->setDataStore(std::move(inCoreStore)); + } + if(edges->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(edgeTupleShape, ShapeType{2}, std::optional{}); + edges->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + edges->setDataStore(std::move(inCoreStore)); + } + } + + if(m_ArrayHandlingType == ArrayHandlingType::Copy) + { std::shared_ptr vertexCopy = vertices->deepCopy(getCreatedPath().createChildPath(m_SharedVerticesName)); const auto vertexArray = std::dynamic_pointer_cast(vertexCopy); @@ -156,8 +177,6 @@ class CreateGeometry1DAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Move) { - edgeTupleShape = edges->getTupleShape(); - vertexTupleShape = vertices->getTupleShape(); const auto geomId = geometry1d->getId(); const auto verticesId = vertices->getId(); @@ -185,8 +204,6 @@ class CreateGeometry1DAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Reference) { - edgeTupleShape = edges->getTupleShape(); - vertexTupleShape = vertices->getTupleShape(); const auto geomId = geometry1d->getId(); dataStructure.setAdditionalParent(vertices->getId(), geomId); dataStructure.setAdditionalParent(edges->getId(), geomId); @@ -198,7 +215,7 @@ class CreateGeometry1DAction : public IDataCreationAction DataPath edgesPath = getCreatedPath().createChildPath(m_SharedEdgesName); // Create the default DataArray that will hold the EdgeList and Vertices. We // size these to 1 because the Csv parser will resize them to the appropriate number of tuples - Result result = ArrayCreationUtilities::CreateArray(dataStructure, edgeTupleShape, {2}, edgesPath, mode, m_CreatedDataStoreFormat); + Result result = ArrayCreationUtilities::CreateArray(dataStructure, edgeTupleShape, {2}, edgesPath, mode); if(result.invalid()) { return MergeResults(result, MakeErrorResult(-5409, fmt::format("{}CreateGeometry1DAction: Could not allocate SharedEdgeList '{}'", prefix, edgesPath.toString()))); @@ -213,7 +230,7 @@ class CreateGeometry1DAction : public IDataCreationAction // Create the Vertex Array with a component size of 3 DataPath vertexPath = getCreatedPath().createChildPath(m_SharedVerticesName); - result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, mode, m_CreatedDataStoreFormat); + result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, mode); if(result.invalid()) { return MergeResults(result, MakeErrorResult(-5410, fmt::format("{}CreateGeometry1DAction: Could not allocate SharedVertList '{}'", prefix, vertexPath.toString()))); @@ -332,7 +349,6 @@ class CreateGeometry1DAction : public IDataCreationAction DataPath m_InputVertices; DataPath m_InputEdges; ArrayHandlingType m_ArrayHandlingType = ArrayHandlingType::Create; - std::string m_CreatedDataStoreFormat; }; using CreateEdgeGeometryAction = CreateGeometry1DAction; diff --git a/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp b/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp index ba1cca68d9..58aa0495e2 100644 --- a/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp +++ b/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp @@ -3,9 +3,11 @@ #include "simplnx/Common/Array.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/Geometry/IGeometry.hpp" #include "simplnx/DataStructure/Geometry/QuadGeom.hpp" #include "simplnx/DataStructure/Geometry/TriangleGeom.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Filter/Output.hpp" #include "simplnx/Utilities/ArrayCreationUtilities.hpp" #include "simplnx/simplnx_export.hpp" @@ -36,7 +38,7 @@ class CreateGeometry2DAction : public IDataCreationAction * @param sharedFacesName The name of the shared face list array to be created */ CreateGeometry2DAction(const DataPath& geometryPath, size_t numFaces, size_t numVertices, const std::string& vertexAttributeMatrixName, const std::string& faceAttributeMatrixName, - const std::string& sharedVerticesName, const std::string& sharedFacesName, std::string createdDataFormat = "") + const std::string& sharedVerticesName, const std::string& sharedFacesName) : IDataCreationAction(geometryPath) , m_NumFaces(numFaces) , m_NumVertices(numVertices) @@ -44,7 +46,6 @@ class CreateGeometry2DAction : public IDataCreationAction , m_FaceDataName(faceAttributeMatrixName) , m_SharedVerticesName(sharedVerticesName) , m_SharedFacesName(sharedFacesName) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -58,7 +59,7 @@ class CreateGeometry2DAction : public IDataCreationAction * @param arrayType Tells whether to copy, move, or reference the existing input vertices array */ CreateGeometry2DAction(const DataPath& geometryPath, const DataPath& inputVerticesArrayPath, const DataPath& inputFacesArrayPath, const std::string& vertexAttributeMatrixName, - const std::string& faceAttributeMatrixName, const ArrayHandlingType& arrayType, std::string createdDataFormat = "") + const std::string& faceAttributeMatrixName, const ArrayHandlingType& arrayType) : IDataCreationAction(geometryPath) , m_VertexDataName(vertexAttributeMatrixName) , m_FaceDataName(faceAttributeMatrixName) @@ -67,7 +68,6 @@ class CreateGeometry2DAction : public IDataCreationAction , m_InputVertices(inputVerticesArrayPath) , m_InputFaces(inputFacesArrayPath) , m_ArrayHandlingType(arrayType) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -140,11 +140,32 @@ class CreateGeometry2DAction : public IDataCreationAction DimensionType faceTupleShape = {m_NumFaces}; DimensionType vertexTupleShape = {m_NumVertices}; // We probably don't know how many Vertices there are but take what ever the developer sends us - if(m_ArrayHandlingType == ArrayHandlingType::Copy) + // For Copy/Move/Reference, read shapes and materialize OOC stores upfront + if(m_ArrayHandlingType != ArrayHandlingType::Create) { faceTupleShape = faces->getTupleShape(); vertexTupleShape = vertices->getTupleShape(); + // If the source arrays have OOC-backed stores, materialize them into + // in-core stores. These arrays may have been created OOC earlier in + // the pipeline when they lived outside any geometry. Unstructured/poly + // geometry topology arrays must be in-core for the visualization layer. + if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(vertexTupleShape, ShapeType{3}, std::optional{}); + vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + vertices->setDataStore(std::move(inCoreStore)); + } + if(faces->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(faceTupleShape, ShapeType{Geometry2DType::k_NumVerts}, std::optional{}); + faces->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + faces->setDataStore(std::move(inCoreStore)); + } + } + + if(m_ArrayHandlingType == ArrayHandlingType::Copy) + { std::shared_ptr vertexCopy = vertices->deepCopy(getCreatedPath().createChildPath(m_SharedVerticesName)); const auto vertexArray = std::dynamic_pointer_cast(vertexCopy); @@ -156,8 +177,6 @@ class CreateGeometry2DAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Move) { - faceTupleShape = faces->getTupleShape(); - vertexTupleShape = vertices->getTupleShape(); const auto geomId = geometry2d->getId(); const auto verticesId = vertices->getId(); @@ -185,8 +204,6 @@ class CreateGeometry2DAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Reference) { - faceTupleShape = faces->getTupleShape(); - vertexTupleShape = vertices->getTupleShape(); const auto geomId = geometry2d->getId(); dataStructure.setAdditionalParent(vertices->getId(), geomId); dataStructure.setAdditionalParent(faces->getId(), geomId); @@ -198,7 +215,7 @@ class CreateGeometry2DAction : public IDataCreationAction DataPath trianglesPath = getCreatedPath().createChildPath(m_SharedFacesName); // Create the default DataArray that will hold the FaceList and Vertices. We // size these to 1 because the Csv parser will resize them to the appropriate number of tuples - Result result = ArrayCreationUtilities::CreateArray(dataStructure, faceTupleShape, {Geometry2DType::k_NumVerts}, trianglesPath, mode, m_CreatedDataStoreFormat); + Result result = ArrayCreationUtilities::CreateArray(dataStructure, faceTupleShape, {Geometry2DType::k_NumVerts}, trianglesPath, mode); if(result.invalid()) { return MergeResults(result, MakeErrorResult(-5509, fmt::format("{}CreateGeometry2DAction: Could not allocate SharedTriList '{}'", prefix, trianglesPath.toString()))); @@ -213,7 +230,7 @@ class CreateGeometry2DAction : public IDataCreationAction // Create the Vertex Array with a component size of 3 DataPath vertexPath = getCreatedPath().createChildPath(m_SharedVerticesName); - result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, mode, m_CreatedDataStoreFormat); + result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, mode); if(result.invalid()) { return MergeResults(result, MakeErrorResult(-5510, fmt::format("{}CreateGeometry2DAction: Could not allocate SharedVertList '{}'", prefix, vertexPath.toString()))); @@ -332,7 +349,6 @@ class CreateGeometry2DAction : public IDataCreationAction DataPath m_InputVertices; DataPath m_InputFaces; ArrayHandlingType m_ArrayHandlingType = ArrayHandlingType::Create; - std::string m_CreatedDataStoreFormat; }; using CreateTriangleGeometryAction = CreateGeometry2DAction; diff --git a/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp b/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp index d944fab566..b60c4a96ed 100644 --- a/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp +++ b/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp @@ -3,9 +3,11 @@ #include "simplnx/Common/Array.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/Geometry/HexahedralGeom.hpp" #include "simplnx/DataStructure/Geometry/IGeometry.hpp" #include "simplnx/DataStructure/Geometry/TetrahedralGeom.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Filter/Output.hpp" #include "simplnx/Utilities/ArrayCreationUtilities.hpp" #include "simplnx/simplnx_export.hpp" @@ -36,7 +38,7 @@ class CreateGeometry3DAction : public IDataCreationAction * @param sharedCellsName The name of the shared cell list array to be created */ CreateGeometry3DAction(const DataPath& geometryPath, size_t numCells, size_t numVertices, const std::string& vertexAttributeMatrixName, const std::string& cellAttributeMatrixName, - const std::string& sharedVerticesName, const std::string& sharedCellsName, std::string createdDataFormat = "") + const std::string& sharedVerticesName, const std::string& sharedCellsName) : IDataCreationAction(geometryPath) , m_NumCells(numCells) , m_NumVertices(numVertices) @@ -44,7 +46,6 @@ class CreateGeometry3DAction : public IDataCreationAction , m_CellDataName(cellAttributeMatrixName) , m_SharedVerticesName(sharedVerticesName) , m_SharedCellsName(sharedCellsName) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -58,7 +59,7 @@ class CreateGeometry3DAction : public IDataCreationAction * @param arrayType Tells whether to copy, move, or reference the existing input vertices array */ CreateGeometry3DAction(const DataPath& geometryPath, const DataPath& inputVerticesArrayPath, const DataPath& inputCellsArrayPath, const std::string& vertexAttributeMatrixName, - const std::string& cellAttributeMatrixName, const ArrayHandlingType& arrayType, std::string createdDataFormat = "") + const std::string& cellAttributeMatrixName, const ArrayHandlingType& arrayType) : IDataCreationAction(geometryPath) , m_VertexDataName(vertexAttributeMatrixName) , m_CellDataName(cellAttributeMatrixName) @@ -67,7 +68,6 @@ class CreateGeometry3DAction : public IDataCreationAction , m_InputVertices(inputVerticesArrayPath) , m_InputCells(inputCellsArrayPath) , m_ArrayHandlingType(arrayType) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -140,11 +140,32 @@ class CreateGeometry3DAction : public IDataCreationAction DimensionType cellTupleShape = {m_NumCells}; DimensionType vertexTupleShape = {m_NumVertices}; // We probably don't know how many Vertices there are but take what ever the developer sends us - if(m_ArrayHandlingType == ArrayHandlingType::Copy) + // For Copy/Move/Reference, read shapes and materialize OOC stores upfront + if(m_ArrayHandlingType != ArrayHandlingType::Create) { cellTupleShape = cells->getTupleShape(); vertexTupleShape = vertices->getTupleShape(); + // If the source arrays have OOC-backed stores, materialize them into + // in-core stores. These arrays may have been created OOC earlier in + // the pipeline when they lived outside any geometry. Unstructured/poly + // geometry topology arrays must be in-core for the visualization layer. + if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(vertexTupleShape, ShapeType{3}, std::optional{}); + vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + vertices->setDataStore(std::move(inCoreStore)); + } + if(cells->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(cellTupleShape, ShapeType{Geometry3DType::k_NumVerts}, std::optional{}); + cells->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + cells->setDataStore(std::move(inCoreStore)); + } + } + + if(m_ArrayHandlingType == ArrayHandlingType::Copy) + { std::shared_ptr vertexCopy = vertices->deepCopy(getCreatedPath().createChildPath(m_SharedVerticesName)); const auto vertexArray = std::dynamic_pointer_cast(vertexCopy); @@ -156,8 +177,6 @@ class CreateGeometry3DAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Move) { - cellTupleShape = cells->getTupleShape(); - vertexTupleShape = vertices->getTupleShape(); const auto geomId = geometry3d->getId(); const auto verticesId = vertices->getId(); @@ -185,8 +204,6 @@ class CreateGeometry3DAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Reference) { - cellTupleShape = cells->getTupleShape(); - vertexTupleShape = vertices->getTupleShape(); const auto geomId = geometry3d->getId(); dataStructure.setAdditionalParent(vertices->getId(), geomId); dataStructure.setAdditionalParent(cells->getId(), geomId); @@ -197,7 +214,7 @@ class CreateGeometry3DAction : public IDataCreationAction { const DataPath cellsPath = getCreatedPath().createChildPath(m_SharedCellsName); // Create the default DataArray that will hold the CellList and Vertices. - Result result = ArrayCreationUtilities::CreateArray(dataStructure, cellTupleShape, {Geometry3DType::k_NumVerts}, cellsPath, mode, m_CreatedDataStoreFormat); + Result result = ArrayCreationUtilities::CreateArray(dataStructure, cellTupleShape, {Geometry3DType::k_NumVerts}, cellsPath, mode); if(result.invalid()) { return MergeResults(result, MakeErrorResult(-5609, fmt::format("{}CreateGeometry3DAction: Could not allocate SharedCellList '{}'", prefix, cellsPath.toString()))); @@ -212,7 +229,7 @@ class CreateGeometry3DAction : public IDataCreationAction // Create the Vertex Array with a component size of 3 const DataPath vertexPath = getCreatedPath().createChildPath(m_SharedVerticesName); - result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, mode, m_CreatedDataStoreFormat); + result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, mode); if(result.invalid()) { return MergeResults(result, MakeErrorResult(-5610, fmt::format("{}CreateGeometry3DAction: Could not allocate SharedVertList '{}'", prefix, vertexPath.toString()))); @@ -331,7 +348,6 @@ class CreateGeometry3DAction : public IDataCreationAction DataPath m_InputVertices; DataPath m_InputCells; ArrayHandlingType m_ArrayHandlingType = ArrayHandlingType::Create; - std::string m_CreatedDataStoreFormat; }; using CreateTetrahedralGeometryAction = CreateGeometry3DAction; diff --git a/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.cpp b/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.cpp index b4706181d3..189d5c9a26 100644 --- a/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.cpp +++ b/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.cpp @@ -10,7 +10,7 @@ namespace nx::core { CreateRectGridGeometryAction::CreateRectGridGeometryAction(const DataPath& path, usize xBoundTuples, usize yBoundTuples, usize zBoundTuples, const std::string& cellAttributeMatrixName, - const std::string& xBoundsName, const std::string& yBoundsName, const std::string& zBoundsName, std::string createdDataFormat) + const std::string& xBoundsName, const std::string& yBoundsName, const std::string& zBoundsName) : IDataCreationAction(path) , m_NumXBoundTuples(xBoundTuples) , m_NumYBoundTuples(yBoundTuples) @@ -19,12 +19,11 @@ CreateRectGridGeometryAction::CreateRectGridGeometryAction(const DataPath& path, , m_XBoundsArrayName(xBoundsName) , m_YBoundsArrayName(yBoundsName) , m_ZBoundsArrayName(zBoundsName) -, m_CreatedDataStoreFormat(createdDataFormat) { } CreateRectGridGeometryAction::CreateRectGridGeometryAction(const DataPath& path, const DataPath& inputXBoundsPath, const DataPath& inputYBoundsPath, const DataPath& inputZBoundsPath, - const std::string& cellAttributeMatrixName, const ArrayHandlingType& arrayType, std::string createdDataFormat) + const std::string& cellAttributeMatrixName, const ArrayHandlingType& arrayType) : IDataCreationAction(path) , m_CellDataName(cellAttributeMatrixName) , m_XBoundsArrayName(inputXBoundsPath.getTargetName()) @@ -34,7 +33,6 @@ CreateRectGridGeometryAction::CreateRectGridGeometryAction(const DataPath& path, , m_InputYBounds(inputYBoundsPath) , m_InputZBounds(inputZBoundsPath) , m_ArrayHandlingType(arrayType) -, m_CreatedDataStoreFormat(createdDataFormat) { } @@ -185,7 +183,7 @@ Float32Array* CreateRectGridGeometryAction::createBoundArray(DataStructure& data { const DimensionType componentShape = {1}; const DataPath boundsPath = getCreatedPath().createChildPath(arrayName); - if(Result<> result = ArrayCreationUtilities::CreateArray(dataStructure, {numTuples}, componentShape, boundsPath, mode, m_CreatedDataStoreFormat); result.invalid()) + if(Result<> result = ArrayCreationUtilities::CreateArray(dataStructure, {numTuples}, componentShape, boundsPath, mode); result.invalid()) { errors.insert(errors.end(), result.errors().begin(), result.errors().end()); return nullptr; diff --git a/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.hpp b/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.hpp index 47eb5d831f..0844af81fb 100644 --- a/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.hpp +++ b/src/simplnx/Filter/Actions/CreateRectGridGeometryAction.hpp @@ -29,7 +29,7 @@ class SIMPLNX_EXPORT CreateRectGridGeometryAction : public IDataCreationAction * @param zBoundsName The name of the zBounds array to be created */ CreateRectGridGeometryAction(const DataPath& path, usize xBoundsDim, usize yBoundsDim, usize zBoundsDim, const std::string& cellAttributeMatrixName, const std::string& xBoundsName, - const std::string& yBoundsName, const std::string& zBoundsName, std::string createdDataFormat = ""); + const std::string& yBoundsName, const std::string& zBoundsName); /** * @brief Constructor to create the geometry using existing x, y, and z bounds arrays by either copying, moving, or referencing them @@ -41,7 +41,7 @@ class SIMPLNX_EXPORT CreateRectGridGeometryAction : public IDataCreationAction * @param arrayType Tells whether to copy, move, or reference the existing input bounds arrays */ CreateRectGridGeometryAction(const DataPath& path, const DataPath& inputXBoundsPath, const DataPath& inputYBoundsPath, const DataPath& inputZBoundsPath, const std::string& cellAttributeMatrixName, - const ArrayHandlingType& arrayType, std::string createdDataFormat = ""); + const ArrayHandlingType& arrayType); ~CreateRectGridGeometryAction() noexcept override; @@ -109,7 +109,6 @@ class SIMPLNX_EXPORT CreateRectGridGeometryAction : public IDataCreationAction DataPath m_InputYBounds; DataPath m_InputZBounds; ArrayHandlingType m_ArrayHandlingType = ArrayHandlingType::Create; - std::string m_CreatedDataStoreFormat; Float32Array* createBoundArray(DataStructure& dataStructure, Mode mode, const std::string& arrayName, usize numTuples, std::vector& errors) const; }; diff --git a/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp b/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp index 25a8a8d4fe..372857bdf5 100644 --- a/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp +++ b/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp @@ -3,8 +3,10 @@ #include "simplnx/Common/Array.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/Geometry/IGeometry.hpp" #include "simplnx/DataStructure/Geometry/VertexGeom.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Filter/Output.hpp" #include "simplnx/Utilities/ArrayCreationUtilities.hpp" #include "simplnx/simplnx_export.hpp" @@ -29,13 +31,11 @@ class CreateVertexGeometryAction : public IDataCreationAction * @param vertexAttributeMatrixName The name of the vertex AttributeMatrix to be created * @param sharedVertexListName The name of the shared vertex list array to be created */ - CreateVertexGeometryAction(const DataPath& geometryPath, IGeometry::MeshIndexType numVertices, const std::string& vertexAttributeMatrixName, const std::string& sharedVertexListName, - std::string createdDataFormat = "") + CreateVertexGeometryAction(const DataPath& geometryPath, IGeometry::MeshIndexType numVertices, const std::string& vertexAttributeMatrixName, const std::string& sharedVertexListName) : IDataCreationAction(geometryPath) , m_NumVertices(numVertices) , m_VertexDataName(vertexAttributeMatrixName) , m_SharedVertexListName(sharedVertexListName) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -46,14 +46,12 @@ class CreateVertexGeometryAction : public IDataCreationAction * @param vertexAttributeMatrixName The name of the vertex AttributeMatrix to be created * @param arrayType Tells whether to copy, move, or reference the existing input vertices array */ - CreateVertexGeometryAction(const DataPath& geometryPath, const DataPath& inputVerticesArrayPath, const std::string& vertexAttributeMatrixName, const ArrayHandlingType& arrayType, - std::string createdDataFormat = "") + CreateVertexGeometryAction(const DataPath& geometryPath, const DataPath& inputVerticesArrayPath, const std::string& vertexAttributeMatrixName, const ArrayHandlingType& arrayType) : IDataCreationAction(geometryPath) , m_VertexDataName(vertexAttributeMatrixName) , m_SharedVertexListName(inputVerticesArrayPath.getTargetName()) , m_InputVertices(inputVerticesArrayPath) , m_ArrayHandlingType(arrayType) - , m_CreatedDataStoreFormat(createdDataFormat) { } @@ -114,11 +112,26 @@ class CreateVertexGeometryAction : public IDataCreationAction ShapeType tupleShape = {m_NumVertices}; // We don't probably know how many Vertices there are but take what ever the developer sends us - // Create the Vertex Array with a component size of 3 - if(m_ArrayHandlingType == ArrayHandlingType::Copy) + // For Copy/Move/Reference, read shapes and materialize OOC stores upfront + if(m_ArrayHandlingType != ArrayHandlingType::Create) { tupleShape = vertices->getTupleShape(); + // If the source array has an OOC-backed store, materialize it into + // an in-core store. The array may have been created OOC earlier in + // the pipeline when it lived outside any geometry. Unstructured/poly + // geometry topology arrays must be in-core for the visualization layer. + if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) + { + auto inCoreStore = std::make_shared>(tupleShape, ShapeType{3}, std::optional{}); + vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + vertices->setDataStore(std::move(inCoreStore)); + } + } + + // Create the Vertex Array with a component size of 3 + if(m_ArrayHandlingType == ArrayHandlingType::Copy) + { std::shared_ptr copy = vertices->deepCopy(getCreatedPath().createChildPath(m_SharedVertexListName)); const auto vertexArray = std::dynamic_pointer_cast(copy); @@ -126,7 +139,6 @@ class CreateVertexGeometryAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Move) { - tupleShape = vertices->getTupleShape(); const auto geomId = vertexGeom->getId(); const auto verticesId = vertices->getId(); dataStructure.setAdditionalParent(verticesId, geomId); @@ -141,7 +153,6 @@ class CreateVertexGeometryAction : public IDataCreationAction } else if(m_ArrayHandlingType == ArrayHandlingType::Reference) { - tupleShape = vertices->getTupleShape(); dataStructure.setAdditionalParent(vertices->getId(), vertexGeom->getId()); vertexGeom->setVertices(*vertices); } @@ -150,7 +161,7 @@ class CreateVertexGeometryAction : public IDataCreationAction const DataPath vertexPath = getCreatedPath().createChildPath(m_SharedVertexListName); const ShapeType componentShape = {3}; - Result<> result = ArrayCreationUtilities::CreateArray(dataStructure, tupleShape, componentShape, vertexPath, mode, m_CreatedDataStoreFormat); + Result<> result = ArrayCreationUtilities::CreateArray(dataStructure, tupleShape, componentShape, vertexPath, mode); if(result.invalid()) { return result; @@ -246,7 +257,6 @@ class CreateVertexGeometryAction : public IDataCreationAction std::string m_SharedVertexListName; DataPath m_InputVertices; ArrayHandlingType m_ArrayHandlingType = ArrayHandlingType::Create; - std::string m_CreatedDataStoreFormat; }; } // namespace nx::core diff --git a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp index 58731528d5..aaeff3b8bc 100644 --- a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp +++ b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp @@ -1,5 +1,7 @@ #include "ImportH5ObjectPathsAction.hpp" +#include "simplnx/DataStructure/IDataArray.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -86,6 +88,82 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod if(useDeferredLoad) { auto ioCollection = DataStoreUtilities::GetIOCollection(); + + // The format resolver is the single decision point for whether an array + // uses in-core or OOC storage. It is registered by the SimplnxOoc plugin + // and considers parent geometry type, user preferences, and data size. + // If no resolver is registered, all arrays default to in-core. + // + // Arrays under unstructured/poly geometries are always resolved to in-core + // because OOC support for those geometry types has been deferred. See the + // resolver implementation in SimplnxOoc for the full rationale. + const auto allPaths = dataStructure.getAllDataPaths(); + + for(const auto& childPath : allPaths) + { + // Only process paths that are children of our imported paths + bool isChild = false; + for(const auto& importedPath : m_Paths) + { + if(childPath.getLength() >= importedPath.getLength()) + { + const auto& ipVec = importedPath.getPathVector(); + const auto& cpVec = childPath.getPathVector(); + bool match = true; + for(usize i = 0; i < ipVec.size(); ++i) + { + if(ipVec[i] != cpVec[i]) + { + match = false; + break; + } + } + if(match) + { + isChild = true; + break; + } + } + } + if(!isChild) + { + continue; + } + + auto* dataObj = dataStructure.getData(childPath); + if(dataObj == nullptr) + { + continue; + } + auto* dataArray = dynamic_cast(dataObj); + if(dataArray == nullptr) + { + continue; + } + const auto* store = dataArray->getIDataStore(); + if(store == nullptr || store->getStoreType() != IDataStore::StoreType::Empty) + { + continue; + } + + // Ask the format resolver what this array should be + uint64 dataSizeBytes = static_cast(store->getNumberOfTuples()) * static_cast(store->getNumberOfComponents()) * static_cast(store->getTypeSize()); + std::string format = ioCollection->resolveFormat(dataStructure, childPath, dataArray->getDataType(), dataSizeBytes); + + if(format.empty()) + { + // Resolver says in-core: eager-load this array from the HDF5 file + auto result = DREAM3D::FinishImportingObject(importStructure, dataStructure, childPath, fileReader, false); + if(result.invalid()) + { + return result; + } + } + // else: leave as Empty for the backfill handler to convert to OOC + } + + // Run the backfill handler on the original paths — it will only find + // arrays that still have Empty stores (the ones the resolver said are OOC) auto backfillResult = ioCollection->runBackfillHandler(dataStructure, m_Paths, fileReader); if(backfillResult.invalid()) { diff --git a/src/simplnx/Utilities/ArrayCreationUtilities.cpp b/src/simplnx/Utilities/ArrayCreationUtilities.cpp index 1476106c44..10d929b4f3 100644 --- a/src/simplnx/Utilities/ArrayCreationUtilities.cpp +++ b/src/simplnx/Utilities/ArrayCreationUtilities.cpp @@ -1,41 +1,13 @@ #include "ArrayCreationUtilities.hpp" -#include "simplnx/Core/Application.hpp" #include "simplnx/Utilities/MemoryUtilities.hpp" using namespace nx::core; //----------------------------------------------------------------------------- -bool ArrayCreationUtilities::CheckMemoryRequirement(DataStructure& dataStructure, uint64 requiredMemory, std::string& format) +bool ArrayCreationUtilities::CheckMemoryRequirement(const DataStructure& dataStructure, uint64 requiredMemory) { static const uint64 k_AvailableMemory = Memory::GetTotalMemory(); - - // If a format is already specified (and it's not the in-memory sentinel), skip the check. - if(!format.empty() && format != Preferences::k_InMemoryFormat) - { - return true; - } - - Preferences* preferencesPtr = Application::GetOrCreateInstance()->getPreferences(); - const uint64 memoryUsage = dataStructure.memoryUsage() + requiredMemory; - const uint64 largeDataStructureSize = preferencesPtr->largeDataStructureSize(); - const std::string largeDataFormat = preferencesPtr->largeDataFormat(); - - // Check if OOC is available: format must be non-empty and not the in-memory sentinel. - bool oocAvailable = !largeDataFormat.empty() && largeDataFormat != Preferences::k_InMemoryFormat; - - if(memoryUsage >= largeDataStructureSize) - { - if(!oocAvailable && memoryUsage >= k_AvailableMemory) - { - return false; - } - if(oocAvailable) - { - format = largeDataFormat; - } - } - - return true; + return memoryUsage < k_AvailableMemory; } diff --git a/src/simplnx/Utilities/ArrayCreationUtilities.hpp b/src/simplnx/Utilities/ArrayCreationUtilities.hpp index 0767672056..998032b802 100644 --- a/src/simplnx/Utilities/ArrayCreationUtilities.hpp +++ b/src/simplnx/Utilities/ArrayCreationUtilities.hpp @@ -19,9 +19,7 @@ namespace nx::core::ArrayCreationUtilities { -inline static constexpr StringLiteral k_DefaultDataFormat = ""; - -SIMPLNX_EXPORT bool CheckMemoryRequirement(DataStructure& dataStructure, uint64 requiredMemory, std::string& format); +SIMPLNX_EXPORT bool CheckMemoryRequirement(const DataStructure& dataStructure, uint64 requiredMemory); /** * @brief Creates a DataArray with the given properties @@ -34,8 +32,7 @@ SIMPLNX_EXPORT bool CheckMemoryRequirement(DataStructure& dataStructure, uint64 * @return */ template -Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, const ShapeType& compShape, const DataPath& path, IDataAction::Mode mode, std::string dataFormat = "", - std::string fillValue = "") +Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, const ShapeType& compShape, const DataPath& path, IDataAction::Mode mode, std::string fillValue = "") { auto parentPath = path.getParent(); @@ -75,18 +72,40 @@ Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, const usize numTuples = std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>()); uint64 requiredMemory = numTuples * numComponents * sizeof(T); - if(!CheckMemoryRequirement(dataStructure, requiredMemory, dataFormat)) + + // Resolve the storage format through the registered hook (e.g., SimplnxOoc). + // The format resolver is the single decision point for whether an array uses + // in-core or OOC storage. It is registered by the SimplnxOoc plugin and + // considers parent geometry type, user preferences, and data size. If no + // resolver is registered, all arrays default to in-core. + // + // Arrays under unstructured/poly geometries are always resolved to in-core + // because OOC support for those geometry types has been deferred. See the + // resolver implementation in SimplnxOoc for the full rationale. + std::string resolvedFormat; + if(mode == IDataAction::Mode::Execute) { - uint64 totalMemory = requiredMemory + dataStructure.memoryUsage(); - uint64 availableMemory = Memory::GetTotalMemory(); - return MakeErrorResult(-264, fmt::format("CreateArray: Cannot create DataArray '{}'.\n\tTotal memory required for DataStructure: '{}' Bytes.\n\tTotal reported memory: '{}' Bytes", name, - totalMemory, availableMemory)); + auto ioCollection = DataStoreUtilities::GetIOCollection(); + resolvedFormat = ioCollection->resolveFormat(dataStructure, path, GetDataType(), requiredMemory); + + // Only check RAM availability for in-core arrays. OOC arrays go to disk + // and do not consume RAM for their primary storage. + if(resolvedFormat.empty() && !CheckMemoryRequirement(dataStructure, requiredMemory)) + { + uint64 totalMemory = requiredMemory + dataStructure.memoryUsage(); + uint64 availableMemory = Memory::GetTotalMemory(); + return MakeErrorResult(-264, fmt::format("Cannot create array '{}': the DataStructure would require {} bytes total, " + "but only {} bytes of RAM are available. Consider enabling out-of-core " + "storage or lowering the size thresholds in Preferences so that large " + "arrays are stored on disk instead of in memory.", + path.toString(), totalMemory, availableMemory)); + } } - auto store = DataStoreUtilities::CreateDataStore(tupleShape, compShape, mode, dataFormat); + auto store = DataStoreUtilities::CreateDataStore(tupleShape, compShape, mode, resolvedFormat); if(nullptr == store) { - return MakeErrorResult(-265, fmt::format("CreateArray: Unable to create DataStore at '{}' of DataStore format '{}'", path.toString(), dataFormat)); + return MakeErrorResult(-265, fmt::format("CreateArray: Unable to create DataStore at '{}' of DataStore format '{}'", path.toString(), resolvedFormat)); } if(!fillValue.empty()) { diff --git a/src/simplnx/Utilities/DataStoreUtilities.hpp b/src/simplnx/Utilities/DataStoreUtilities.hpp index 057c59e2f1..a9072bf44d 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.hpp +++ b/src/simplnx/Utilities/DataStoreUtilities.hpp @@ -27,54 +27,34 @@ uint64 CalculateDataSize(const ShapeType& tupleShape, const ShapeType& component } /** - * @brief Creates a DataStore with the given properties, choosing between in-memory - * and out-of-core storage based on the active format resolver. + * @brief Simple factory that creates a DataStore with the given properties. * - * In Preflight mode, returns an EmptyDataStore that records shape metadata - * without allocating any storage. In Execute mode, the function consults the - * DataIOCollection's format resolver to determine the appropriate storage - * format (e.g., "HDF5-OOC" for out-of-core, or "" for in-memory). If the - * caller provides an explicit dataFormat, the resolver is bypassed. + * This function does NOT resolve the storage format. The caller is responsible + * for determining the correct format (e.g., by calling the format resolver in + * CreateArray) and passing it in via the dataFormat parameter. * - * The format resolution flow in Execute mode is: - * 1. If dataFormat is non-empty, use it as-is (caller override). - * 2. Otherwise, call ioCollection->resolveFormat() which asks the registered - * plugin (e.g., SimplnxOoc) whether this array should be OOC based on - * its type, shape, and total byte size. - * 3. Pass the resolved format to createDataStoreWithType(), which creates - * either an in-memory DataStore (for "" or k_InMemoryFormat) or an - * OOC-backed store (for "HDF5-OOC" etc.). + * In Preflight mode, returns an EmptyDataStore that records shape metadata + * without allocating any storage. In Execute mode, forwards directly to + * createDataStoreWithType() which creates either an in-memory DataStore + * (for "" or k_InMemoryFormat) or an OOC-backed store (for "HDF5-OOC" etc.). * * @tparam T Primitive type (int8, float32, uint64, etc.) * @param tupleShape The tuple dimensions (e.g., {100, 200, 300} for a 3D volume) * @param componentShape The component dimensions (e.g., {3} for a 3-component vector) * @param mode PREFLIGHT returns an EmptyDataStore; EXECUTE allocates real storage - * @param dataFormat Optional explicit format name. If empty, the format resolver decides. + * @param dataFormat The already-resolved format name. Empty string means in-memory. * @return Shared pointer to the created AbstractDataStore */ template -std::shared_ptr> CreateDataStore(const ShapeType& tupleShape, const ShapeType& componentShape, IDataAction::Mode mode, std::string dataFormat = "") +std::shared_ptr> CreateDataStore(const ShapeType& tupleShape, const ShapeType& componentShape, IDataAction::Mode mode, const std::string& dataFormat = "") { switch(mode) { case IDataAction::Mode::Preflight: { - // Preflight: no storage allocated, just record the shape and format hint return std::make_unique>(tupleShape, componentShape, dataFormat); } case IDataAction::Mode::Execute: { - // Compute the total byte size so the format resolver can make size-based decisions - uint64 dataSize = CalculateDataSize(tupleShape, componentShape); auto ioCollection = GetIOCollection(); - - // If no explicit format was requested, ask the registered resolver (if any) - // to determine whether this array should be in-memory or OOC - if(dataFormat.empty()) - { - dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, componentShape, dataSize); - } - - // Create the store using the resolved format. The IOCollection dispatches - // to the appropriate IDataIOManager based on the format string. return ioCollection->createDataStoreWithType(dataFormat, tupleShape, componentShape); } default: { @@ -84,22 +64,25 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap } /** - * @brief Creates a ListStore (backing store for NeighborList arrays) with the - * given properties, using the same format resolution as CreateDataStore. + * @brief Simple factory that creates a ListStore with the given properties. + * + * This function does NOT resolve the storage format. The caller is responsible + * for determining the correct format (e.g., by calling the format resolver in + * CreateNeighborListAction) and passing it in via the dataFormat parameter. * - * NeighborLists have variable-length per-tuple data, so the actual byte size - * is not known at creation time. The function estimates the size using a - * component count of 10 as a heuristic, which is sufficient for the format - * resolver to decide between in-memory and OOC storage. + * In Preflight mode, returns an EmptyListStore that records shape metadata + * without allocating any storage. In Execute mode, forwards directly to + * createListStoreWithType() which creates either an in-memory ListStore + * (for "" or k_InMemoryFormat) or an OOC-backed store (for "HDF5-OOC" etc.). * * @tparam T Primitive type of the list elements * @param tupleShape The tuple dimensions * @param mode PREFLIGHT returns an EmptyListStore; EXECUTE allocates real storage - * @param dataFormat Optional explicit format name. If empty, the format resolver decides. + * @param dataFormat The already-resolved format name. Empty string means in-memory. * @return Shared pointer to the created AbstractListStore */ template -std::shared_ptr> CreateListStore(const ShapeType& tupleShape, IDataAction::Mode mode = IDataAction::Mode::Execute, std::string dataFormat = "") +std::shared_ptr> CreateListStore(const ShapeType& tupleShape, IDataAction::Mode mode = IDataAction::Mode::Execute, const std::string& dataFormat = "") { switch(mode) { @@ -108,14 +91,7 @@ std::shared_ptr> CreateListStore(const ShapeType& tupleShap return std::make_unique>(tupleShape); } case IDataAction::Mode::Execute: { - // Estimate byte size with a heuristic component count of 10 for format resolution. - // The actual per-tuple list lengths are unknown until data is populated. - uint64 dataSize = CalculateDataSize(tupleShape, {10}); auto ioCollection = GetIOCollection(); - if(dataFormat.empty()) - { - dataFormat = ioCollection->resolveFormat(GetDataType(), tupleShape, {10}, dataSize); - } return ioCollection->createListStoreWithType(dataFormat, tupleShape); } default: { diff --git a/test/DataIOCollectionHooksTest.cpp b/test/DataIOCollectionHooksTest.cpp index 45b5fdf7b2..0565758fba 100644 --- a/test/DataIOCollectionHooksTest.cpp +++ b/test/DataIOCollectionHooksTest.cpp @@ -1,3 +1,4 @@ +#include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" #include @@ -10,31 +11,37 @@ TEST_CASE("DataIOCollectionHooks: format resolver default behavior") REQUIRE(collection.hasFormatResolver() == false); // With no resolver set, should return empty string - std::string result = collection.resolveFormat(DataType::float32, {100, 100, 100}, {1}, 4000000); + DataStructure ds; + DataPath dp({"TestArray"}); + std::string result = collection.resolveFormat(ds, dp, DataType::float32, 4000000); REQUIRE(result.empty()); } TEST_CASE("DataIOCollectionHooks: format resolver returns plugin result") { DataIOCollection collection; - collection.setFormatResolver([](DataType, const ShapeType&, const ShapeType&, uint64 sizeBytes) -> std::string { return sizeBytes > 1000 ? "HDF5-OOC" : ""; }); + collection.setFormatResolver([](const DataStructure&, const DataPath&, DataType, uint64 sizeBytes) -> std::string { return sizeBytes > 1000 ? "HDF5-OOC" : ""; }); REQUIRE(collection.hasFormatResolver() == true); + DataStructure ds; + DataPath dp({"TestArray"}); // Small array - returns "" - REQUIRE(collection.resolveFormat(DataType::float32, {10}, {1}, 40).empty()); + REQUIRE(collection.resolveFormat(ds, dp, DataType::float32, 40).empty()); // Large array - returns "HDF5-OOC" - REQUIRE(collection.resolveFormat(DataType::float32, {1000}, {1}, 4000) == "HDF5-OOC"); + REQUIRE(collection.resolveFormat(ds, dp, DataType::float32, 4000) == "HDF5-OOC"); } TEST_CASE("DataIOCollectionHooks: format resolver can be unset") { DataIOCollection collection; - collection.setFormatResolver([](DataType, const ShapeType&, const ShapeType&, uint64) -> std::string { return "HDF5-OOC"; }); + collection.setFormatResolver([](const DataStructure&, const DataPath&, DataType, uint64) -> std::string { return "HDF5-OOC"; }); REQUIRE(collection.hasFormatResolver() == true); collection.setFormatResolver(nullptr); REQUIRE(collection.hasFormatResolver() == false); - REQUIRE(collection.resolveFormat(DataType::float32, {10}, {1}, 40).empty()); + DataStructure ds; + DataPath dp({"TestArray"}); + REQUIRE(collection.resolveFormat(ds, dp, DataType::float32, 40).empty()); } TEST_CASE("DataIOCollectionHooks: backfill handler default behavior") diff --git a/test/IOFormat.cpp b/test/IOFormat.cpp index 2ba0f5c37e..2b54479c66 100644 --- a/test/IOFormat.cpp +++ b/test/IOFormat.cpp @@ -1,6 +1,7 @@ #include #include "simplnx/Core/Application.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" #include "simplnx/DataStructure/IO/Generic/IDataIOManager.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" @@ -116,8 +117,10 @@ TEST_CASE("Data Format: resolveFormat returns empty with no resolver registered" prefs->setLargeDataFormat(std::string(Preferences::k_InMemoryFormat)); // With no OOC plugin loaded, no resolver is registered and resolveFormat returns "" + DataStructure ds; + DataPath dp({"TestArray"}); uint64 largeSize = 2ULL * 1024 * 1024 * 1024; // 2 GB — well above any threshold - std::string dataFormat = ioCollection->resolveFormat(DataType::float32, {100, 100, 100}, {1}, largeSize); + std::string dataFormat = ioCollection->resolveFormat(ds, dp, DataType::float32, largeSize); REQUIRE(dataFormat.empty()); // Should NOT be changed to any OOC format prefs->setLargeDataFormat(savedFormat); @@ -133,8 +136,10 @@ TEST_CASE("Data Format: resolveFormat returns empty when format not configured", // Empty format = not configured, and no OOC plugin is loaded in the in-core build prefs->setLargeDataFormat(""); + DataStructure ds; + DataPath dp({"TestArray"}); uint64 largeSize = 2ULL * 1024 * 1024 * 1024; - std::string dataFormat = ioCollection->resolveFormat(DataType::float32, {100, 100, 100}, {1}, largeSize); + std::string dataFormat = ioCollection->resolveFormat(ds, dp, DataType::float32, largeSize); REQUIRE(dataFormat.empty()); // No OOC format available prefs->setLargeDataFormat(savedFormat); From 054ff2f843a0787fa758a18ccb36476150ba30e2 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Thu, 9 Apr 2026 14:44:17 -0400 Subject: [PATCH 04/13] =?UTF-8?q?REFACTOR:=20Recovery=20reattachment=20ref?= =?UTF-8?q?actor=20=E2=80=94=20rename,=20restructure,=20and=20harden=20.dr?= =?UTF-8?q?eam3d=20import?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename the "backfill handler" to "data store import handler" and expand its role to handle ALL data store loading from .dream3d files — in-core eager loading, OOC reference stores, and recovery reattachment. This replaces the split decision-making where ImportH5ObjectPathsAction ran a format-resolver loop and a separate backfill handler. Key changes: 1. DataIOCollection: Rename BackfillHandlerFnc to DataStoreImportHandlerFnc with expanded signature that includes importStructure. Rename set/has/runBackfillHandler to set/has/runDataStoreImportHandler. Add format display name registry (registerFormatDisplayName/getFormatDisplayNames) for human-readable format names in the UI dropdown. 2. DataStoreIO: Rename ReadDataStore to ReadDataStoreIntoMemory. Remove recovery reattachment code (OOC-specific HDF5 attribute checks moved to SimplnxOoc plugin). Add placeholder detection — compares physical HDF5 element count against shape attributes, returns Result<> with warning when mismatch detected (guards against loading placeholder datasets without the OOC plugin). Change return type to Result>> so callers can accumulate warnings across arrays. 3. ImportH5ObjectPathsAction: Remove the format-resolver loop (79 lines). The action now delegates entirely to the registered handler when present, or falls back to FinishImportingObject for non-OOC builds. 4. CreateArrayAction: Restore dataFormat parameter for per-filter format override. When non-empty, bypasses the format resolver. Dropdown shows "Automatic" (resolver decides), "In Memory", and plugin-registered formats with display names. Fix 12 filter callers where fillValue was being passed as dataFormat after parameter reordering. 5. Dream3dIO: Route DREAM3D::ReadFile through ImportH5ObjectPathsAction so recovery and OOC hooks fire. Remove unused ImportDataObjectFromFile and ImportSelectDataObjectsFromFile. 6. Application: Add getDataStoreFormatDisplayNames() to expose display name registry to DataStoreFormatParameter. Updated callers: DataArrayIO (2 sites), NeighborListIO (2 sites), Dream3dIO (2 legacy helpers), DataStructureWriter (comment), 12 filter files, simplnxpy Python binding, DataIOCollectionHooksTest. --- .../Filters/ComputeFZQuaternionsFilter.cpp | 2 +- .../Filters/AppendImageGeometryFilter.cpp | 2 +- .../Filters/ComputeFeatureNeighborsFilter.cpp | 2 +- .../Filters/ComputeKMeansFilter.cpp | 5 +- .../Filters/ComputeKMedoidsFilter.cpp | 5 +- .../Filters/ComputeSurfaceFeaturesFilter.cpp | 2 +- .../Filters/ComputeVectorColorsFilter.cpp | 2 +- .../Filters/CreateDataArrayAdvancedFilter.cpp | 4 +- .../Filters/CreateDataArrayFilter.cpp | 4 +- ...eateFeatureArrayFromElementArrayFilter.cpp | 2 +- .../src/SimplnxCore/Filters/DBSCANFilter.cpp | 5 +- .../MapPointCloudToRegularGridFilter.cpp | 2 +- .../Filters/ReadTextDataArrayFilter.cpp | 4 +- .../Filters/ScalarSegmentFeaturesFilter.cpp | 4 +- .../SimplnxCore/Filters/SilhouetteFilter.cpp | 2 +- .../SimplnxCore/wrapping/python/simplnxpy.cpp | 4 +- src/simplnx/Core/Application.cpp | 5 + src/simplnx/Core/Application.hpp | 12 ++ .../IO/Generic/DataIOCollection.cpp | 42 ++++-- .../IO/Generic/DataIOCollection.hpp | 100 ++++++++----- .../DataStructure/IO/HDF5/DataArrayIO.hpp | 69 ++++++--- .../DataStructure/IO/HDF5/DataStoreIO.hpp | 135 ++++-------------- .../IO/HDF5/DataStructureWriter.cpp | 2 +- .../DataStructure/IO/HDF5/NeighborListIO.hpp | 54 +++++-- .../Filter/Actions/CreateArrayAction.cpp | 19 ++- .../Filter/Actions/CreateArrayAction.hpp | 17 ++- .../Actions/ImportH5ObjectPathsAction.cpp | 89 +----------- .../Parameters/DataStoreFormatParameter.cpp | 21 ++- .../Parameters/DataStoreFormatParameter.hpp | 16 ++- .../Utilities/ArrayCreationUtilities.hpp | 27 +++- .../Utilities/Parsing/DREAM3D/Dream3dIO.cpp | 99 ++++++------- .../Utilities/Parsing/DREAM3D/Dream3dIO.hpp | 4 - test/DataIOCollectionHooksTest.cpp | 4 +- 33 files changed, 408 insertions(+), 358 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp index 54cc97f6a5..d7d1d4603d 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeFZQuaternionsFilter.cpp @@ -123,7 +123,7 @@ IFilter::PreflightResult ComputeFZQuaternionsFilter::preflightImpl(const DataStr nx::core::Result resultOutputActions; auto createArrayAction = - std::make_unique(nx::core::DataType::float32, quatArray.getDataStore()->getTupleShape(), quatArray.getDataStore()->getComponentShape(), pFZQuatsArrayPathValue, "0.0"); + std::make_unique(nx::core::DataType::float32, quatArray.getDataStore()->getTupleShape(), quatArray.getDataStore()->getComponentShape(), pFZQuatsArrayPathValue, "", "0.0"); resultOutputActions.value().appendAction(std::move(createArrayAction)); // Return both the resultOutputActions and the preflightUpdatedValues via std::move() diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp index 7c9c80d3f3..a26106d072 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/AppendImageGeometryFilter.cpp @@ -307,7 +307,7 @@ IFilter::PreflightResult AppendImageGeometryFilter::preflightImpl(const DataStru auto compShape = inputDataArray != nullptr ? inputDataArray->getComponentShape() : destDataArray->getComponentShape(); auto cellDataDims = pSaveAsNewGeometry ? newCellDataDims : destCellDataDims; auto cellArrayPath = pSaveAsNewGeometry ? newArrayPath : destArrayPath; - auto createArrayAction = std::make_unique(dataType, cellDataDims, compShape, cellArrayPath, pDefaultValue); + auto createArrayAction = std::make_unique(dataType, cellDataDims, compShape, cellArrayPath, "", pDefaultValue); resultOutputActions.value().appendAction(std::move(createArrayAction)); } if(arrayType == IArray::ArrayType::NeighborListArray) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp index 183dc57d7f..b518d3d59d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp @@ -158,7 +158,7 @@ IFilter::PreflightResult ComputeFeatureNeighborsFilter::preflightImpl(const Data // Create the SurfaceFeatures Output Data Array in the Feature Attribute Matrix if(storeSurfaceFeatures) { - auto action = std::make_unique(DataType::boolean, tupleShape, cDims, surfaceFeaturesPath, "false"); + auto action = std::make_unique(DataType::boolean, tupleShape, cDims, surfaceFeaturesPath, "", "false"); actions.appendAction(std::move(action)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp index abab2c1296..5bea16d31f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMeansFilter.cpp @@ -137,7 +137,8 @@ IFilter::PreflightResult ComputeKMeansFilter::preflightImpl(const DataStructure& } { - auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "0"); + auto createAction = + std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "", "0"); resultOutputActions.value().appendAction(std::move(createAction)); } @@ -145,7 +146,7 @@ IFilter::PreflightResult ComputeKMeansFilter::preflightImpl(const DataStructure& { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "", "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp index 0bc5b16754..556944f6c8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp @@ -132,7 +132,8 @@ IFilter::PreflightResult ComputeKMedoidsFilter::preflightImpl(const DataStructur } { - auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "0"); + auto createAction = + std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "", "0"); resultOutputActions.value().appendAction(std::move(createAction)); } @@ -140,7 +141,7 @@ IFilter::PreflightResult ComputeKMedoidsFilter::preflightImpl(const DataStructur { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "", "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp index 75067685f2..f9808f0c63 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp @@ -110,7 +110,7 @@ IFilter::PreflightResult ComputeSurfaceFeaturesFilter::preflightImpl(const DataS } auto createSurfaceFeaturesAction = - std::make_unique(DataType::uint8, tupleDims, std::vector{1}, pCellFeaturesAttributeMatrixPathValue.createChildPath(pSurfaceFeaturesArrayNameValue), "0"); + std::make_unique(DataType::uint8, tupleDims, std::vector{1}, pCellFeaturesAttributeMatrixPathValue.createChildPath(pSurfaceFeaturesArrayNameValue), "", "0"); resultOutputActions.value().appendAction(std::move(createSurfaceFeaturesAction)); return {std::move(resultOutputActions), std::move(preflightUpdatedValues)}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp index 939170d438..44451da0e2 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeVectorColorsFilter.cpp @@ -109,7 +109,7 @@ IFilter::PreflightResult ComputeVectorColorsFilter::preflightImpl(const DataStru if(!pUseGoodVoxelsValue) { - auto action = std::make_unique(DataType::boolean, vectorsTupShape, std::vector{1}, k_MaskArrayPath, "true"); + auto action = std::make_unique(DataType::boolean, vectorsTupShape, std::vector{1}, k_MaskArrayPath, "", "true"); resultOutputActions.value().appendAction(std::move(action)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp index 7785b8f414..4b94916513 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayAdvancedFilter.cpp @@ -220,8 +220,10 @@ IFilter::PreflightResult CreateDataArrayAdvancedFilter::preflightImpl(const Data usize numTuples = std::accumulate(tupleDims.begin(), tupleDims.end(), static_cast(1), std::multiplies<>()); + auto dataFormat = filterArgs.value(k_DataFormat_Key); + auto arrayDataType = ConvertNumericTypeToDataType(numericType); - auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath); + auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath, dataFormat); resultOutputActions.value().appendAction(std::move(action)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp index ccd4217974..6d29ddb265 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateDataArrayFilter.cpp @@ -154,8 +154,10 @@ IFilter::PreflightResult CreateDataArrayFilter::preflightImpl(const DataStructur } } + auto dataFormat = filterArgs.value(k_DataFormat_Key); + // Sanity check that init value can be converted safely to the final numeric type integrated into action - auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath, initValue); + auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, compDims, dataArrayPath, dataFormat, initValue); resultOutputActions.value().appendAction(std::move(action)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp index ba31a15ea4..ae061a3d82 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/CreateFeatureArrayFromElementArrayFilter.cpp @@ -105,7 +105,7 @@ IFilter::PreflightResult CreateFeatureArrayFromElementArrayFilter::preflightImpl { DataType dataType = selectedCellArray.getDataType(); auto createArrayAction = - std::make_unique(dataType, amTupleShape, selectedCellArrayStore.getComponentShape(), pCellFeatureAttributeMatrixPathValue.createChildPath(pCreatedArrayNameValue), "0"); + std::make_unique(dataType, amTupleShape, selectedCellArrayStore.getComponentShape(), pCellFeatureAttributeMatrixPathValue.createChildPath(pCreatedArrayNameValue), "", "0"); resultOutputActions.value().appendAction(std::move(createArrayAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp index e49ffd81b8..c378d5a743 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp @@ -160,7 +160,8 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt } { - auto createAction = std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "0"); + auto createAction = + std::make_unique(DataType::int32, clusterArray->getTupleShape(), std::vector{1}, pSelectedArrayPathValue.replaceName(pFeatureIdsArrayNameValue), "", "0"); resultOutputActions.value().appendAction(std::move(createAction)); } @@ -168,7 +169,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "", "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp index f09bdf078e..6dfab6932a 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MapPointCloudToRegularGridFilter.cpp @@ -351,7 +351,7 @@ IFilter::PreflightResult MapPointCloudToRegularGridFilter::preflightImpl(const D { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, vertexData->getShape(), std::vector{1}, tempPath, "true"); + auto createAction = std::make_unique(DataType::boolean, vertexData->getShape(), std::vector{1}, tempPath, "", "true"); actions.appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp index 77e36d9c96..b81408efa8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadTextDataArrayFilter.cpp @@ -137,7 +137,9 @@ IFilter::PreflightResult ReadTextDataArrayFilter::preflightImpl(const DataStruct } } - auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, std::vector{nComp}, arrayPath); + auto dataFormat = filterArgs.value(k_DataFormat_Key); + + auto action = std::make_unique(ConvertNumericTypeToDataType(numericType), tupleDims, std::vector{nComp}, arrayPath, dataFormat); resultOutputActions.value().appendAction(std::move(action)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp index 2b64ac60eb..f461b2245f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ScalarSegmentFeaturesFilter.cpp @@ -137,11 +137,11 @@ IFilter::PreflightResult ScalarSegmentFeaturesFilter::preflightImpl(const DataSt } // Create the Cell Level FeatureIds array - auto createFeatureIdsAction = std::make_unique(DataType::int32, cellTupleDims, std::vector{1}, featureIdsPath, "0"); + auto createFeatureIdsAction = std::make_unique(DataType::int32, cellTupleDims, std::vector{1}, featureIdsPath, "", "0"); // Create the Feature Attribute Matrix auto createFeatureGroupAction = std::make_unique(cellFeaturesPath, std::vector{1}); - auto createActiveAction = std::make_unique(DataType::uint8, std::vector{1}, std::vector{1}, activeArrayPath, "1"); + auto createActiveAction = std::make_unique(DataType::uint8, std::vector{1}, std::vector{1}, activeArrayPath, "", "1"); nx::core::Result resultOutputActions; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp index b2753a2aff..1774efbb8b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SilhouetteFilter.cpp @@ -120,7 +120,7 @@ IFilter::PreflightResult SilhouetteFilter::preflightImpl(const DataStructure& da { DataPath tempPath = DataPath({k_MaskName}); { - auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "true"); + auto createAction = std::make_unique(DataType::boolean, clusterArray->getTupleShape(), std::vector{1}, tempPath, "", "true"); resultOutputActions.value().appendAction(std::move(createAction)); } diff --git a/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp b/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp index bea05d76cd..f7c8d27a1d 100644 --- a/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp +++ b/src/Plugins/SimplnxCore/wrapping/python/simplnxpy.cpp @@ -1174,8 +1174,8 @@ PYBIND11_MODULE(simplnx, mod) copyDataObjectAction.def(py::init>(), "path"_a, "new_path"_a, "all_created_paths"_a); auto createArrayAction = SIMPLNX_PY_BIND_CLASS_VARIADIC(mod, CreateArrayAction, IDataCreationAction); - createArrayAction.def(py::init&, const std::vector&, const DataPath&, std::string>(), "type"_a, "t_dims"_a, "c_dims"_a, "path"_a, - "fill_value"_a = std::string("")); + createArrayAction.def(py::init&, const std::vector&, const DataPath&, std::string, std::string>(), "type"_a, "t_dims"_a, "c_dims"_a, "path"_a, + "data_format"_a = std::string(""), "fill_value"_a = std::string("")); auto createAttributeMatrixAction = SIMPLNX_PY_BIND_CLASS_VARIADIC(mod, CreateAttributeMatrixAction, IDataCreationAction); createAttributeMatrixAction.def(py::init(), "path"_a, "shape"_a); diff --git a/src/simplnx/Core/Application.cpp b/src/simplnx/Core/Application.cpp index ebd49eab85..8a28cd855b 100644 --- a/src/simplnx/Core/Application.cpp +++ b/src/simplnx/Core/Application.cpp @@ -414,3 +414,8 @@ std::vector Application::getDataStoreFormats() const { return m_DataIOCollection->getFormatNames(); } + +std::vector> Application::getDataStoreFormatDisplayNames() const +{ + return m_DataIOCollection->getFormatDisplayNames(); +} diff --git a/src/simplnx/Core/Application.hpp b/src/simplnx/Core/Application.hpp index 7f9d727b0f..f96939e6b6 100644 --- a/src/simplnx/Core/Application.hpp +++ b/src/simplnx/Core/Application.hpp @@ -10,6 +10,7 @@ #include #include #include +#include #include namespace nx::core @@ -190,6 +191,17 @@ class SIMPLNX_EXPORT Application */ std::vector getDataStoreFormats() const; + /** + * @brief Returns all known format display names as (formatName, displayName) pairs. + * + * Delegates to DataIOCollection::getFormatDisplayNames(). The list always + * includes ("", "Automatic") and (k_InMemoryFormat, "In Memory"), plus any + * plugin-registered entries. + * + * @return Vector of (formatName, displayName) pairs + */ + std::vector> getDataStoreFormatDisplayNames() const; + protected: /** * @brief Constructs an Application using default values and replaces the diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index 388d790e8b..17d9a4fa0a 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -235,32 +235,31 @@ std::string DataIOCollection::resolveFormat(const DataStructure& dataStructure, } // --------------------------------------------------------------------------- -// Backfill handler hook (replaces placeholders after .dream3d import) +// Data store import handler hook (loads all data stores after .dream3d import) // --------------------------------------------------------------------------- -void DataIOCollection::setBackfillHandler(BackfillHandlerFnc handler) +void DataIOCollection::setDataStoreImportHandler(DataStoreImportHandlerFnc handler) { - // Store the callback. An empty std::function effectively disables backfill. - m_BackfillHandler = std::move(handler); + // Store the callback. An empty std::function effectively disables the handler. + m_DataStoreImportHandler = std::move(handler); } -bool DataIOCollection::hasBackfillHandler() const +bool DataIOCollection::hasDataStoreImportHandler() const { - return static_cast(m_BackfillHandler); + return static_cast(m_DataStoreImportHandler); } -Result<> DataIOCollection::runBackfillHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader) +Result<> DataIOCollection::runDataStoreImportHandler(DataStructure& importStructure, DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader) { // If no handler is installed, this is the in-memory code path: placeholders // will be replaced elsewhere (or eagerly loaded). Return success. - if(!m_BackfillHandler) + if(!m_DataStoreImportHandler) { return {}; } - // Delegate to the plugin-provided handler, which replaces placeholder stores - // (EmptyDataStore, EmptyListStore, EmptyStringStore) with real stores - // (typically OOC read-only references pointing at the same HDF5 file). - return m_BackfillHandler(dataStructure, paths, fileReader); + // Delegate to the plugin-provided handler, which loads all data stores — + // in-core, OOC, or recovered — for the imported paths. + return m_DataStoreImportHandler(importStructure, dataStructure, paths, fileReader); } std::vector DataIOCollection::getFormatNames() const @@ -277,6 +276,25 @@ std::vector DataIOCollection::getFormatNames() const return keyNames; } +void DataIOCollection::registerFormatDisplayName(const std::string& formatName, const std::string& displayName) +{ + m_FormatDisplayNames[formatName] = displayName; +} + +std::vector> DataIOCollection::getFormatDisplayNames() const +{ + std::vector> result; + // Always include the two built-in entries first + result.emplace_back("", "Automatic"); + result.emplace_back(std::string(Preferences::k_InMemoryFormat), "In Memory"); + // Append any plugin-registered display names + for(const auto& [formatName, displayName] : m_FormatDisplayNames) + { + result.emplace_back(formatName, displayName); + } + return result; +} + DataIOCollection::iterator DataIOCollection::begin() { return m_ManagerMap.begin(); diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index 42933fd5bc..c126e09e2e 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -67,27 +67,29 @@ class SIMPLNX_EXPORT DataIOCollection using FormatResolverFnc = std::function; /** - * @brief Callback invoked once after a .dream3d file has been imported with - * placeholder (empty) stores. + * @brief Callback responsible for ALL data store loading from .dream3d files — + * in-core, OOC, and recovery. * * During import, the HDF5 reader creates lightweight placeholder stores * (EmptyDataStore, EmptyListStore, EmptyStringStore) for every array so that - * the DataStructure's topology is complete without loading data. The backfill - * handler is then called with the list of imported DataPaths and the still-open - * HDF5 file reader, giving the OOC plugin the opportunity to replace each - * placeholder with a real store -- typically a read-only OOC reference that - * lazily reads chunks from the same file. - * - * Only one backfill handler may be registered at a time. If no handler is - * registered, the placeholders remain (which is valid for in-memory workflows - * that eagerly load data through a different path). - * - * @param dataStructure The DataStructure containing placeholder stores to backfill - * @param paths DataPaths of the objects imported from the file - * @param fileReader Open HDF5 file reader for the source .dream3d file + * the DataStructure's topology is complete without loading data. The data + * store import handler is then called with the list of imported DataPaths and + * the still-open HDF5 file reader, giving the OOC plugin the opportunity to + * replace each placeholder with a real store — either a read-only OOC + * reference that lazily reads chunks from the same file, an eagerly-loaded + * in-core store, or a recovered store from a prior session. + * + * Only one data store import handler may be registered at a time. If no + * handler is registered, the placeholders remain (which is valid for + * in-memory workflows that eagerly load data through a different path). + * + * @param importStructure The intermediate DataStructure from the HDF5 reader (holds reader factories) + * @param dataStructure The target DataStructure containing placeholder stores to replace + * @param paths DataPaths of the objects imported from the file + * @param fileReader Open HDF5 file reader for the source .dream3d file * @return Result<> indicating success or describing any errors */ - using BackfillHandlerFnc = std::function(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader)>; + using DataStoreImportHandlerFnc = std::function(DataStructure& importStructure, DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader)>; /** * @brief Callback that can intercept and override how a DataObject is written @@ -158,36 +160,38 @@ class SIMPLNX_EXPORT DataIOCollection std::string resolveFormat(const DataStructure& dataStructure, const DataPath& arrayPath, DataType numericType, uint64 dataSizeBytes) const; /** - * @brief Registers or clears the backfill handler callback. + * @brief Registers or clears the data store import handler callback. * * The OOC plugin calls this once during loading to install a handler that - * replaces placeholder stores with OOC reference stores after a .dream3d - * file is imported. Passing an empty std::function disables the handler. + * replaces placeholder stores with real stores (in-core, OOC, or recovered) + * after a .dream3d file is imported. Passing an empty std::function disables + * the handler. * - * @param handler The backfill handler callback, or an empty std::function to clear it + * @param handler The data store import handler callback, or an empty std::function to clear it */ - void setBackfillHandler(BackfillHandlerFnc handler); + void setDataStoreImportHandler(DataStoreImportHandlerFnc handler); /** - * @brief Checks whether a backfill handler callback is currently registered. + * @brief Checks whether a data store import handler callback is currently registered. * @return true if a non-empty handler callback is set */ - bool hasBackfillHandler() const; + bool hasDataStoreImportHandler() const; /** - * @brief Invokes the registered backfill handler to replace placeholder stores - * with real stores after importing a .dream3d file. + * @brief Invokes the registered data store import handler to replace placeholder + * stores with real stores after importing a .dream3d file. * * If no handler is registered, this is a no-op and returns an empty success - * Result. This allows the in-memory code path to skip backfill entirely while - * the OOC code path processes every imported array. + * Result. This allows the in-memory code path to skip store import entirely + * while the OOC code path processes every imported array. * - * @param dataStructure The DataStructure whose placeholder stores should be replaced - * @param paths DataPaths of the objects imported from the file - * @param fileReader Open HDF5 file reader for the source .dream3d file + * @param importStructure The intermediate DataStructure from the HDF5 reader (holds reader factories) + * @param dataStructure The target DataStructure whose placeholder stores should be replaced + * @param paths DataPaths of the objects imported from the file + * @param fileReader Open HDF5 file reader for the source .dream3d file * @return Result<> indicating success, or containing errors from the handler */ - Result<> runBackfillHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader); + Result<> runDataStoreImportHandler(DataStructure& importStructure, DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader); /** * @brief Registers or clears the write-array-override callback. @@ -448,12 +452,38 @@ class SIMPLNX_EXPORT DataIOCollection */ const_iterator end() const; + /** + * @brief Registers a human-readable display name for a data store format. + * + * Plugins call this during hook registration to associate their internal + * format name (e.g., "HDF5-OOC") with a user-friendly label (e.g., + * "HDF5 Out-of-Core") for display in the DataStoreFormatParameter dropdown. + * + * @param formatName The internal format identifier + * @param displayName The human-readable label shown in the UI + */ + void registerFormatDisplayName(const std::string& formatName, const std::string& displayName); + + /** + * @brief Returns all known format display names as (formatName, displayName) pairs. + * + * The returned list always starts with: + * - ("", "Automatic") -- lets the resolver decide + * - (Preferences::k_InMemoryFormat, "In Memory") -- explicit in-memory + * + * Followed by any plugin-registered entries (e.g., ("HDF5-OOC", "HDF5 Out-of-Core")). + * + * @return Vector of (formatName, displayName) pairs + */ + std::vector> getFormatDisplayNames() const; + private: map_type m_ManagerMap; - FormatResolverFnc m_FormatResolver; ///< Plugin-provided callback that selects storage format for new arrays - BackfillHandlerFnc m_BackfillHandler; ///< Plugin-provided callback that replaces placeholders after .dream3d import - WriteArrayOverrideFnc m_WriteArrayOverride; ///< Plugin-provided callback that intercepts DataObject writes to HDF5 - bool m_WriteArrayOverrideActive = false; ///< Gate flag: override fires only when both registered and active + FormatResolverFnc m_FormatResolver; ///< Plugin-provided callback that selects storage format for new arrays + DataStoreImportHandlerFnc m_DataStoreImportHandler; ///< Plugin-provided callback that loads all data stores after .dream3d import + WriteArrayOverrideFnc m_WriteArrayOverride; ///< Plugin-provided callback that intercepts DataObject writes to HDF5 + bool m_WriteArrayOverrideActive = false; ///< Gate flag: override fires only when both registered and active + std::map m_FormatDisplayNames; ///< Plugin-registered human-readable format names }; /** diff --git a/src/simplnx/DataStructure/IO/HDF5/DataArrayIO.hpp b/src/simplnx/DataStructure/IO/HDF5/DataArrayIO.hpp index 0f2db650e1..b6245759c8 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataArrayIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataArrayIO.hpp @@ -39,11 +39,28 @@ class DataArrayIO : public IDataIO */ template static void importDataArray(DataStructure& dataStructure, const nx::core::HDF5::DatasetIO& datasetReader, const std::string dataArrayName, DataObject::IdType importId, - nx::core::HDF5::ErrorType& err, const std::optional& parentId, bool preflight) + nx::core::HDF5::ErrorType& err, const std::optional& parentId, bool preflight, std::vector& warnings) { - std::shared_ptr> dataStore = - preflight ? std::shared_ptr>(EmptyDataStoreIO::ReadDataStore(datasetReader)) : (DataStoreIO::ReadDataStore(datasetReader)); - DataArray* data = DataArray::Import(dataStructure, dataArrayName, importId, std::move(dataStore), parentId); + if(preflight) + { + std::shared_ptr> dataStore(EmptyDataStoreIO::ReadDataStore(datasetReader)); + DataArray* data = DataArray::Import(dataStructure, dataArrayName, importId, std::move(dataStore), parentId); + err = (data == nullptr) ? -400 : 0; + return; + } + + auto storeResult = DataStoreIO::ReadDataStoreIntoMemory(datasetReader); + for(auto&& warning : storeResult.warnings()) + { + warnings.push_back(std::move(warning)); + } + if(storeResult.value() == nullptr) + { + // Placeholder detected — skip this array without error + err = 0; + return; + } + DataArray* data = DataArray::Import(dataStructure, dataArrayName, importId, std::move(storeResult.value()), parentId); err = (data == nullptr) ? -400 : 0; } @@ -58,13 +75,16 @@ class DataArrayIO : public IDataIO template static Result<> importDataStore(data_type* dataArray, const DataPath& dataPath, const nx::core::HDF5::DatasetIO& datasetReader) { - std::shared_ptr> dataStore = DataStoreIO::ReadDataStore(datasetReader); - if(dataStore == nullptr) + auto storeResult = DataStoreIO::ReadDataStoreIntoMemory(datasetReader); + Result<> result; + result.m_Warnings = std::move(storeResult.warnings()); + if(storeResult.value() == nullptr) { - return MakeErrorResult(-150202, fmt::format("Failed to import DataArray data at path '{}'.", dataPath.toString())); + // Placeholder detected — propagate warnings, skip without error + return result; } - dataArray->setDataStore(dataStore); - return {}; + dataArray->setDataStore(std::move(storeResult.value())); + return result; } /** @@ -167,45 +187,46 @@ class DataArrayIO : public IDataIO } int32 err = 0; + std::vector warnings; switch(type) { case DataType::float32: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::float64: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::int8: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::int16: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::int32: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::int64: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::uint8: { if(isBoolArray) { - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); } else { - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); } } break; case DataType::uint16: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::uint32: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; case DataType::uint64: - importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore); + importDataArray(dataStructureReader.getDataStructure(), datasetReader, dataArrayName, importId, err, parentId, useEmptyDataStore, warnings); break; default: { err = -777; @@ -215,10 +236,14 @@ class DataArrayIO : public IDataIO if(err < 0) { - return MakeErrorResult(err, fmt::format("Error importing dataset from HDF5 file. DataArray name '{}' that is a child of '{}'", dataArrayName, parentGroup.getName())); + auto result = MakeErrorResult(err, fmt::format("Error importing dataset from HDF5 file. DataArray name '{}' that is a child of '{}'", dataArrayName, parentGroup.getName())); + result.m_Warnings = std::move(warnings); + return result; } - return {}; + Result<> result; + result.m_Warnings = std::move(warnings); + return result; } /** diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp index 5135bdf74d..86104438fb 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStoreIO.hpp @@ -1,15 +1,15 @@ #pragma once +#include "simplnx/Common/Result.hpp" #include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/IO/HDF5/IDataStoreIO.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/DatasetIO.hpp" -#include #include -#include -#include +#include +#include namespace nx::core { @@ -48,123 +48,48 @@ inline Result<> WriteDataStore(nx::core::HDF5::DatasetIO& datasetWriter, const A } /** - * @brief Reads a DataStore from an HDF5 dataset, selecting between three - * loading strategies based on recovery-file metadata and OOC preferences. + * @brief Reads an HDF5 dataset into an in-memory DataStore. * - * The three paths, tried in order, are: + * Reads tuple/component shapes from HDF5 attributes, allocates an + * in-core DataStore, and loads all data from the dataset into memory. + * This function does not handle OOC stores or recovery-file placeholders; + * those are handled by the data store import handler at a higher level. * - * 1. **Recovery-file reattach** -- If the dataset carries OocBackingFilePath / - * OocBackingDatasetPath / OocChunkShape string attributes, this is a - * placeholder dataset written by a recovery file. The method reattaches to - * the original backing file via createReadOnlyRefStore(). If the backing - * file is missing or the OOC plugin is not loaded, falls through. + * If the physical HDF5 dataset element count does not match the expected + * count from shape attributes, the dataset is skipped and a warning is + * returned. This guards against reading placeholder datasets written by + * plugins that are not currently loaded. * - * 2. **OOC format resolution** -- For arrays whose byte size exceeds the - * user's largeDataSize threshold (or if force-OOC is enabled), the - * plugin-registered format resolver selects an OOC format string (e.g. - * "HDF5-OOC"). A read-only reference store is created that points at - * the *source* HDF5 file, avoiding a multi-gigabyte copy into a temp - * file. Chunk shapes are derived from the HDF5 dataset's chunk layout; - * for contiguous datasets a default shape is synthesized (one Z-slice - * for 3D data, or clamped-to-64 for 1D/2D data). - * - * 3. **Normal in-core load** -- Creates a standard in-memory DataStore via - * CreateDataStore and reads all data from HDF5 into RAM. - * - * Each OOC path is guarded: if no OOC plugin is loaded (no factory - * registered for the format), createReadOnlyRefStore() returns nullptr and - * the code falls through transparently to the next strategy. - * - * @tparam T The element type of the data store * @param datasetReader The HDF5 dataset to read from - * @return std::shared_ptr> The created data store, - * either in-core or backed by an OOC reference store + * @return Result containing the in-memory data store, or a warning with + * nullptr if the dataset is a placeholder */ template -inline std::shared_ptr> ReadDataStore(const nx::core::HDF5::DatasetIO& datasetReader) +inline Result>> ReadDataStoreIntoMemory(const nx::core::HDF5::DatasetIO& datasetReader) { auto tupleShape = IDataStoreIO::ReadTupleShape(datasetReader); auto componentShape = IDataStoreIO::ReadComponentShape(datasetReader); - // ----------------------------------------------------------------------- - // PATH 1: Recovery-file reattach - // ----------------------------------------------------------------------- - // Recovery files (written by WriteRecoveryFile) store OOC arrays as - // zero-byte scalar placeholder datasets annotated with three string - // attributes: - // - OocBackingFilePath : absolute path to the original backing HDF5 file - // - OocBackingDatasetPath: HDF5 dataset path inside the backing file - // - OocChunkShape : comma-separated chunk dimensions (e.g. "1,200,200") - // - // If these attributes exist and the backing file is still on disk, we - // create a read-only reference store pointing at the backing file instead - // of reading the (empty) placeholder data. This allows recovery files to - // rehydrate OOC arrays without duplicating the data. - // - // No compile-time guard is needed: createReadOnlyRefStore returns nullptr - // if no OOC plugin is loaded, and we fall through to path 2/3. - { - auto backingPathResult = datasetReader.readStringAttribute("OocBackingFilePath"); - if(backingPathResult.valid() && !backingPathResult.value().empty()) - { - std::filesystem::path backingFilePath = backingPathResult.value(); + // Check that the physical HDF5 dataset size matches the expected size + // from shape attributes. A mismatch indicates the dataset is a + // placeholder (e.g. written by an OOC plugin that is not loaded). + usize expectedElements = std::accumulate(tupleShape.cbegin(), tupleShape.cend(), static_cast(1), std::multiplies<>()) * + std::accumulate(componentShape.cbegin(), componentShape.cend(), static_cast(1), std::multiplies<>()); + usize physicalElements = datasetReader.getNumElements(); - if(std::filesystem::exists(backingFilePath)) - { - // Read the HDF5-internal dataset path from the backing file attribute. - auto datasetPathResult = datasetReader.readStringAttribute("OocBackingDatasetPath"); - std::string backingDatasetPath = datasetPathResult.valid() ? datasetPathResult.value() : ""; - - // Parse the chunk shape from a comma-separated string attribute - // (e.g. "1,200,200" for one Z-slice of a 3D dataset). The chunk - // shape tells the OOC store how to partition reads/writes. - auto chunkResult = datasetReader.readStringAttribute("OocChunkShape"); - ShapeType chunkShape; - if(chunkResult.valid() && !chunkResult.value().empty()) - { - std::stringstream ss(chunkResult.value()); - std::string token; - while(std::getline(ss, token, ',')) - { - if(!token.empty()) - { - chunkShape.push_back(static_cast(std::stoull(token))); - } - } - } - - // Ask the IO collection to create a read-only reference store backed - // by the original file. The format is always "HDF5-OOC" for recovery - // files since that is what wrote the placeholder. - auto ioCollection = DataStoreUtilities::GetIOCollection(); - auto refStore = ioCollection->createReadOnlyRefStore("HDF5-OOC", GetDataType(), backingFilePath, backingDatasetPath, tupleShape, componentShape, chunkShape); - if(refStore != nullptr) - { - // Successfully reattached; wrap the IDataStore unique_ptr in a - // shared_ptr> via dynamic_pointer_cast. - return std::shared_ptr>(std::dynamic_pointer_cast>(std::shared_ptr(std::move(refStore)))); - } - // createReadOnlyRefStore returned nullptr (OOC plugin not loaded); - // fall through to path 2/3. - } - else - { - std::cerr << "[RECOVERY] Backing file missing for OOC array: " << backingFilePath << std::endl; - // Fall through to load placeholder data - } - } + if(physicalElements != expectedElements) + { + Result>> result; + result.warnings().push_back(Warning{-89200, fmt::format("Unable to read dataset '{}' at path '{}': the file contains {} elements but the shape " + "attributes indicate {} elements. This typically means the dataset is a placeholder written " + "by a plugin that is not currently loaded. Ensure all required plugins are loaded and try again.", + datasetReader.getName(), datasetReader.getObjectPath(), physicalElements, expectedElements)}); + return result; } - // ----------------------------------------------------------------------- - // PATH 2: Normal in-core load - // ----------------------------------------------------------------------- - // Format resolution for imported data is handled by the backfill strategy - // at a higher level (CreateArray / ImportH5ObjectPathsAction). During the - // eager HDF5 read path, we always load in-core. OOC decisions for imported - // data are made when the DataStructure is populated, not during raw I/O. auto dataStore = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); dataStore->readHdf5(datasetReader); - return dataStore; + return {std::move(dataStore)}; } } // namespace DataStoreIO } // namespace HDF5 diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp index 3df7df23df..85b8d6391c 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp @@ -127,7 +127,7 @@ Result<> DataStructureWriter::writeDataObject(const DataObject* dataObject, nx:: // placeholder dataset annotated with OocBackingFilePath / // OocBackingDatasetPath / OocChunkShape attributes, so the recovery // file stays small while preserving enough metadata to reattach to the - // backing file on reload (see ReadDataStore path 1). + // backing file on reload (see the data store import handler). // // If the callback returns std::nullopt the object is not OOC-backed, so // we fall through to the normal HDF5 write path below. diff --git a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp index 2c7accab4b..25c250cf99 100644 --- a/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp +++ b/src/simplnx/DataStructure/IO/HDF5/NeighborListIO.hpp @@ -39,12 +39,17 @@ class NeighborListIO : public IDataIO * HDF5, split into per-tuple vectors using the NumNeighbors companion * array, and packed into an in-memory ListStore. * + * If the NumNeighbors companion array is a placeholder (element count + * mismatch), warnings are accumulated and nullptr is returned. The caller + * should treat this as a skip, not an error. + * * @param parentGroup The HDF5 group containing the dataset and its companion * @param dataReader The HDF5 dataset containing the flat packed neighbor data * @param useEmptyDataStore If true, return an EmptyListStore placeholder - * @return std::shared_ptr The created list store, or nullptr on error + * @param warnings Output vector to accumulate any warnings encountered + * @return std::shared_ptr The created list store, or nullptr on error/placeholder */ - static std::shared_ptr ReadHdf5Data(const nx::core::HDF5::GroupIO& parentGroup, const nx::core::HDF5::DatasetIO& dataReader, bool useEmptyDataStore = false) + static std::shared_ptr ReadHdf5Data(const nx::core::HDF5::GroupIO& parentGroup, const nx::core::HDF5::DatasetIO& dataReader, bool useEmptyDataStore, std::vector& warnings) { try { @@ -68,8 +73,17 @@ class NeighborListIO : public IDataIO return std::make_shared>(tupleDimsResult.value()); } - auto numNeighborsPtr = DataStoreIO::ReadDataStore(numNeighborsReader); - auto& numNeighborsStore = *numNeighborsPtr.get(); + auto numNeighborsResult = DataStoreIO::ReadDataStoreIntoMemory(numNeighborsReader); + for(auto&& warning : numNeighborsResult.warnings()) + { + warnings.push_back(std::move(warning)); + } + if(numNeighborsResult.value() == nullptr) + { + // NumNeighbors is a placeholder — cannot populate NeighborList + return nullptr; + } + auto& numNeighborsStore = *numNeighborsResult.value(); auto flatDataStorePtr = dataReader.template readAsDataStore(); if(flatDataStorePtr == nullptr) @@ -122,14 +136,25 @@ class NeighborListIO : public IDataIO const std::optional& parentId, bool useEmptyDataStore = false) const override { auto datasetReader = parentGroup.openDataset(objectName); - auto listStorePtr = ReadHdf5Data(parentGroup, datasetReader, useEmptyDataStore); + std::vector warnings; + auto listStorePtr = ReadHdf5Data(parentGroup, datasetReader, useEmptyDataStore, warnings); + + Result<> result; + result.m_Warnings = std::move(warnings); + + if(listStorePtr == nullptr && !result.m_Warnings.empty()) + { + // Placeholder detected — skip this NeighborList, propagate warnings + return result; + } + auto* dataObject = data_type::Import(dataStructureReader.getDataStructure(), objectName, importId, listStorePtr, parentId); if(dataObject == nullptr) { std::string ss = "Failed to import NeighborList from HDF5"; return MakeErrorResult(-505, ss); } - return {}; + return result; } /** @@ -167,8 +192,19 @@ class NeighborListIO : public IDataIO // Read the "NumNeighbors" companion array, which stores the per-tuple // neighbor count used to interpret the flat packed data array. auto numNeighborsReader = parentGroup.openDataset(numNeighborsName); - auto numNeighborsPtr = DataStoreIO::ReadDataStore(numNeighborsReader); - auto& numNeighborsStore = *numNeighborsPtr.get(); + auto numNeighborsResult = DataStoreIO::ReadDataStoreIntoMemory(numNeighborsReader); + + Result<> result; + for(auto&& warning : numNeighborsResult.warnings()) + { + result.m_Warnings.push_back(std::move(warning)); + } + if(numNeighborsResult.value() == nullptr) + { + // NumNeighbors is a placeholder — cannot populate NeighborList, propagate warnings + return result; + } + auto& numNeighborsStore = *numNeighborsResult.value(); const auto numTuples = numNeighborsStore.getNumberOfTuples(); const auto tupleShape = numNeighborsStore.getTupleShape(); @@ -206,7 +242,7 @@ class NeighborListIO : public IDataIO } neighborList.setStore(listStorePtr); - return {}; + return result; } /** diff --git a/src/simplnx/Filter/Actions/CreateArrayAction.cpp b/src/simplnx/Filter/Actions/CreateArrayAction.cpp index 1389d31e46..19b0333986 100644 --- a/src/simplnx/Filter/Actions/CreateArrayAction.cpp +++ b/src/simplnx/Filter/Actions/CreateArrayAction.cpp @@ -10,21 +10,23 @@ namespace struct CreateArrayFunctor { template - Result<> operator()(DataStructure& dataStructure, const std::vector& tDims, const std::vector& cDims, const DataPath& path, IDataAction::Mode mode, std::string fillValue) + Result<> operator()(DataStructure& dataStructure, const std::vector& tDims, const std::vector& cDims, const DataPath& path, IDataAction::Mode mode, const std::string& dataFormat, + std::string fillValue) { - return ArrayCreationUtilities::CreateArray(dataStructure, tDims, cDims, path, mode, fillValue); + return ArrayCreationUtilities::CreateArray(dataStructure, tDims, cDims, path, mode, dataFormat, fillValue); } }; } // namespace namespace nx::core { -CreateArrayAction::CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string fillValue) +CreateArrayAction::CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string dataFormat, std::string fillValue) : IDataCreationAction(path) , m_Type(type) , m_Dims(tDims) , m_CDims(cDims) -, m_FillValue(fillValue) +, m_DataFormat(std::move(dataFormat)) +, m_FillValue(std::move(fillValue)) { } @@ -32,12 +34,12 @@ CreateArrayAction::~CreateArrayAction() noexcept = default; Result<> CreateArrayAction::apply(DataStructure& dataStructure, Mode mode) const { - return ExecuteDataFunction(::CreateArrayFunctor{}, m_Type, dataStructure, m_Dims, m_CDims, getCreatedPath(), mode, m_FillValue); + return ExecuteDataFunction(::CreateArrayFunctor{}, m_Type, dataStructure, m_Dims, m_CDims, getCreatedPath(), mode, m_DataFormat, m_FillValue); } IDataAction::UniquePointer CreateArrayAction::clone() const { - return std::make_unique(m_Type, m_Dims, m_CDims, getCreatedPath()); + return std::make_unique(m_Type, m_Dims, m_CDims, getCreatedPath(), m_DataFormat, m_FillValue); } DataType CreateArrayAction::type() const @@ -69,4 +71,9 @@ std::string CreateArrayAction::fillValue() const { return m_FillValue; } + +std::string CreateArrayAction::dataFormat() const +{ + return m_DataFormat; +} } // namespace nx::core diff --git a/src/simplnx/Filter/Actions/CreateArrayAction.hpp b/src/simplnx/Filter/Actions/CreateArrayAction.hpp index 0d76909977..2d49197427 100644 --- a/src/simplnx/Filter/Actions/CreateArrayAction.hpp +++ b/src/simplnx/Filter/Actions/CreateArrayAction.hpp @@ -26,9 +26,12 @@ class SIMPLNX_EXPORT CreateArrayAction : public IDataCreationAction * @param tDims The tuple dimensions * @param cDims The component dimensions * @param path The path where the DataArray will be created + * @param dataFormat The data store format override. Empty string means "Automatic" + * (let the format resolver decide). A non-empty value bypasses the + * resolver and uses the specified format directly. * @param fillValue The fill value for the array */ - CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string fillValue = ""); + CreateArrayAction(DataType type, const std::vector& tDims, const std::vector& cDims, const DataPath& path, std::string dataFormat = "", std::string fillValue = ""); ~CreateArrayAction() noexcept override; @@ -88,10 +91,22 @@ class SIMPLNX_EXPORT CreateArrayAction : public IDataCreationAction */ std::string fillValue() const; + /** + * @brief Returns the data store format override for this action. + * + * Empty string means "Automatic" -- the format resolver decides. A non-empty + * value bypasses the resolver and uses the specified format directly, allowing + * individual filters to override the global format policy. + * + * @return The data format string + */ + std::string dataFormat() const; + private: DataType m_Type; std::vector m_Dims; std::vector m_CDims; + std::string m_DataFormat = ""; std::string m_FillValue = ""; }; } // namespace nx::core diff --git a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp index aaeff3b8bc..fe81e92115 100644 --- a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp +++ b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp @@ -1,7 +1,5 @@ #include "ImportH5ObjectPathsAction.hpp" -#include "simplnx/DataStructure/IDataArray.hpp" -#include "simplnx/DataStructure/IDataStore.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -56,7 +54,7 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod if(!preflighting) { auto ioCollection = DataStoreUtilities::GetIOCollection(); - useDeferredLoad = ioCollection->hasBackfillHandler(); + useDeferredLoad = ioCollection->hasDataStoreImportHandler(); } std::stringstream errorMessages; @@ -69,7 +67,7 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod // When deferring loading, pass preflight=true to avoid the expensive // data copy. The resulting placeholder stores are finalized by the - // registered backfill handler below. + // registered data store import handler below. auto result = DREAM3D::FinishImportingObject(importStructure, dataStructure, targetPath, fileReader, preflighting || useDeferredLoad); if(result.invalid()) { @@ -84,90 +82,13 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod return MakeErrorResult(-6201, errorMessages.str()); } - // Delegate placeholder finalization to the registered backfill handler (if any) if(useDeferredLoad) { auto ioCollection = DataStoreUtilities::GetIOCollection(); - - // The format resolver is the single decision point for whether an array - // uses in-core or OOC storage. It is registered by the SimplnxOoc plugin - // and considers parent geometry type, user preferences, and data size. - // If no resolver is registered, all arrays default to in-core. - // - // Arrays under unstructured/poly geometries are always resolved to in-core - // because OOC support for those geometry types has been deferred. See the - // resolver implementation in SimplnxOoc for the full rationale. - const auto allPaths = dataStructure.getAllDataPaths(); - - for(const auto& childPath : allPaths) - { - // Only process paths that are children of our imported paths - bool isChild = false; - for(const auto& importedPath : m_Paths) - { - if(childPath.getLength() >= importedPath.getLength()) - { - const auto& ipVec = importedPath.getPathVector(); - const auto& cpVec = childPath.getPathVector(); - bool match = true; - for(usize i = 0; i < ipVec.size(); ++i) - { - if(ipVec[i] != cpVec[i]) - { - match = false; - break; - } - } - if(match) - { - isChild = true; - break; - } - } - } - if(!isChild) - { - continue; - } - - auto* dataObj = dataStructure.getData(childPath); - if(dataObj == nullptr) - { - continue; - } - auto* dataArray = dynamic_cast(dataObj); - if(dataArray == nullptr) - { - continue; - } - const auto* store = dataArray->getIDataStore(); - if(store == nullptr || store->getStoreType() != IDataStore::StoreType::Empty) - { - continue; - } - - // Ask the format resolver what this array should be - uint64 dataSizeBytes = static_cast(store->getNumberOfTuples()) * static_cast(store->getNumberOfComponents()) * static_cast(store->getTypeSize()); - std::string format = ioCollection->resolveFormat(dataStructure, childPath, dataArray->getDataType(), dataSizeBytes); - - if(format.empty()) - { - // Resolver says in-core: eager-load this array from the HDF5 file - auto result = DREAM3D::FinishImportingObject(importStructure, dataStructure, childPath, fileReader, false); - if(result.invalid()) - { - return result; - } - } - // else: leave as Empty for the backfill handler to convert to OOC - } - - // Run the backfill handler on the original paths — it will only find - // arrays that still have Empty stores (the ones the resolver said are OOC) - auto backfillResult = ioCollection->runBackfillHandler(dataStructure, m_Paths, fileReader); - if(backfillResult.invalid()) + auto importResult = ioCollection->runDataStoreImportHandler(importStructure, dataStructure, m_Paths, fileReader); + if(importResult.invalid()) { - return backfillResult; + return importResult; } } diff --git a/src/simplnx/Parameters/DataStoreFormatParameter.cpp b/src/simplnx/Parameters/DataStoreFormatParameter.cpp index a423d1f190..f97e4c59d2 100644 --- a/src/simplnx/Parameters/DataStoreFormatParameter.cpp +++ b/src/simplnx/Parameters/DataStoreFormatParameter.cpp @@ -66,13 +66,30 @@ typename DataStoreFormatParameter::ValueType DataStoreFormatParameter::defaultSt typename DataStoreFormatParameter::AvailableValuesType DataStoreFormatParameter::availableValues() const { - return Application::GetOrCreateInstance()->getDataStoreFormats(); + const auto displayNames = Application::GetOrCreateInstance()->getDataStoreFormatDisplayNames(); + AvailableValuesType result; + result.reserve(displayNames.size()); + for(const auto& [formatName, displayName] : displayNames) + { + result.push_back(formatName); + } + return result; +} + +std::vector> DataStoreFormatParameter::availableFormatsWithDisplayNames() const +{ + return Application::GetOrCreateInstance()->getDataStoreFormatDisplayNames(); } Result<> DataStoreFormatParameter::validate(const std::any& value) const { [[maybe_unused]] const auto& stringValue = GetAnyRef(value); - const auto formats = Application::GetOrCreateInstance()->getDataStoreFormats(); + // Empty string is always valid — it means "Automatic" (let the resolver decide) + if(stringValue.empty()) + { + return {}; + } + const auto formats = availableValues(); if(std::find(formats.begin(), formats.end(), stringValue) == formats.end()) { std::string ss = fmt::format("DataStore format not known: '{}'", stringValue); diff --git a/src/simplnx/Parameters/DataStoreFormatParameter.hpp b/src/simplnx/Parameters/DataStoreFormatParameter.hpp index eeca60a785..ee9e2033b7 100644 --- a/src/simplnx/Parameters/DataStoreFormatParameter.hpp +++ b/src/simplnx/Parameters/DataStoreFormatParameter.hpp @@ -5,6 +5,7 @@ #include "simplnx/simplnx_export.hpp" #include +#include #include namespace nx::core @@ -64,11 +65,22 @@ class SIMPLNX_EXPORT DataStoreFormatParameter : public ValueParameter ValueType defaultString() const; /** - * @brief - * @retrurn + * @brief Returns the list of available format name strings. + * @return Vector of format name strings (keys only, no display names) */ AvailableValuesType availableValues() const; + /** + * @brief Returns all available formats as (formatName, displayName) pairs. + * + * The list always includes ("", "Automatic") and (k_InMemoryFormat, "In Memory"), + * plus any plugin-registered formats. This is intended for UI widgets that need + * to display human-readable labels alongside the internal format identifiers. + * + * @return Vector of (formatName, displayName) pairs + */ + std::vector> availableFormatsWithDisplayNames() const; + /** * @brief * @param value diff --git a/src/simplnx/Utilities/ArrayCreationUtilities.hpp b/src/simplnx/Utilities/ArrayCreationUtilities.hpp index 998032b802..26ef98e794 100644 --- a/src/simplnx/Utilities/ArrayCreationUtilities.hpp +++ b/src/simplnx/Utilities/ArrayCreationUtilities.hpp @@ -3,6 +3,7 @@ #include "simplnx/simplnx_export.hpp" #include "simplnx/Common/Result.hpp" +#include "simplnx/Core/Preferences.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataStructure.hpp" @@ -32,7 +33,8 @@ SIMPLNX_EXPORT bool CheckMemoryRequirement(const DataStructure& dataStructure, u * @return */ template -Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, const ShapeType& compShape, const DataPath& path, IDataAction::Mode mode, std::string fillValue = "") +Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, const ShapeType& compShape, const DataPath& path, IDataAction::Mode mode, const std::string& dataFormat = "", + std::string fillValue = "") { auto parentPath = path.getParent(); @@ -85,8 +87,27 @@ Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, std::string resolvedFormat; if(mode == IDataAction::Mode::Execute) { - auto ioCollection = DataStoreUtilities::GetIOCollection(); - resolvedFormat = ioCollection->resolveFormat(dataStructure, path, GetDataType(), requiredMemory); + if(!dataFormat.empty()) + { + // User explicitly chose a format via the filter UI — bypass the resolver. + // The k_InMemoryFormat sentinel means "force in-memory"; translate it to + // an empty string so the downstream DataStore factory creates an in-core store. + if(dataFormat == Preferences::k_InMemoryFormat.str()) + { + resolvedFormat = ""; + } + else + { + resolvedFormat = dataFormat; + } + } + else + { + // No per-filter override — ask the resolver (which consults user preferences, + // size thresholds, and geometry type). + auto ioCollection = DataStoreUtilities::GetIOCollection(); + resolvedFormat = ioCollection->resolveFormat(dataStructure, path, GetDataType(), requiredMemory); + } // Only check RAM availability for in-core arrays. OOC arrays go to disk // and do not consume RAM for their primary storage. diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp index b3dc17f743..5dbec18d06 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp @@ -23,6 +23,7 @@ #include "simplnx/DataStructure/NeighborList.hpp" #include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/DataStructure/StringStore.hpp" +#include "simplnx/Filter/Actions/ImportH5ObjectPathsAction.hpp" #include "simplnx/Pipeline/Pipeline.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -1120,14 +1121,25 @@ Result<> createLegacyNeighborList(DataStructure& dataStructure, DataObject ::IdT { // Read the NeighborList data from HDF5. In preflight mode, this returns // an empty store with the correct tuple count but no actual list data. - auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader, preflight); + std::vector warnings; + auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader, preflight, warnings); + + Result<> result; + result.m_Warnings = std::move(warnings); + + if(listStore == nullptr && !result.m_Warnings.empty()) + { + // Placeholder detected — skip without error, propagate warnings + return result; + } + auto* neighborList = NeighborList::Create(dataStructure, datasetReader.getName(), listStore, parentId); if(neighborList == nullptr) { std::string ss = fmt::format("Failed to create NeighborList: '{}'", datasetReader.getName()); return MakeErrorResult(Legacy::k_FailedCreatingNeighborList_Code, ss); } - return {}; + return result; } Result<> readLegacyNeighborList(DataStructure& dataStructure, const nx::core::HDF5::GroupIO& parentReader, const nx::core::HDF5::DatasetIO& datasetReader, DataObject::IdType parentId, @@ -1194,13 +1206,23 @@ Result<> finishImportingLegacyNeighborListImpl(DataStructure& dataStructure, con return MakeErrorResult(-4210426, fmt::format("Failed to finish importing legacy NeighborList at path '{}'. Imported NeighborList not found.", dataPath.toString())); } - auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader); + std::vector warnings; + auto listStore = HDF5::NeighborListIO::ReadHdf5Data(parentReader, datasetReader, false, warnings); + + Result<> result; + result.m_Warnings = std::move(warnings); + if(listStore == nullptr) { + if(!result.m_Warnings.empty()) + { + // Placeholder detected — skip without error, propagate warnings + return result; + } return MakeErrorResult(-4210427, fmt::format("Failed to finish importing legacy NeighborList at path '{}'. Failed to import HDF5 data.", dataPath.toString())); } existingList->setStore(listStore); - return {}; + return result; } Result<> finishImportingLegacyNeighborList(DataStructure& dataStructure, const nx::core::HDF5::GroupIO& parentReader, const HDF5::DatasetIO& datasetReader, const DataPath& dataPath) @@ -2292,46 +2314,6 @@ Result DREAM3D::ImportPipelineJsonFromFile(const std::filesystem return ImportPipelineJsonFromFile(fileReader); } -Result> DREAM3D::ImportDataObjectFromFile(const nx::core::HDF5::FileIO& fileReader, const DataPath& dataPath) -{ - const auto fileVersion = GetFileVersion(fileReader); - if(fileVersion == k_CurrentFileVersion) - { - return HDF5::DataStructureReader::ReadObject(fileReader, dataPath); - } - else if(fileVersion == k_LegacyFileVersion) - { - auto result = ImportLegacyDataObjectFromFile(fileReader, dataPath); - if(result.invalid()) - { - return ConvertInvalidResult>(std::move(result)); - } - std::vector> value = result.value(); - if(value.size() != 0) - { - return MakeErrorResult>(-48264, fmt::format("Error extracting a single DataObject from legacy DREAM3D file at path '{}'", dataPath.toString())); - } - return {result.value().front()}; - } - return MakeErrorResult>(-523242, fmt::format("Error extracting a single DataObject from legacy DREAM3D file at path '{}'", dataPath.toString())); -} - -Result>> DREAM3D::ImportSelectDataObjectsFromFile(const nx::core::HDF5::FileIO& fileReader, const std::vector& dataPaths) -{ - std::vector> dataObjects; - for(const DataPath& dataPath : dataPaths) - { - auto importResult = ImportDataObjectFromFile(fileReader, dataPath); - if(importResult.invalid()) - { - return ConvertInvalidResult>>(std::move(importResult)); - } - dataObjects.push_back(std::move(importResult.value())); - } - - return {dataObjects}; -} - Result<> DREAM3D::FinishImportingObject(DataStructure& importStructure, DataStructure& dataStructure, const DataPath& dataPath, const nx::core::HDF5::FileIO& fileReader, bool preflight) { if(!importStructure.containsData(dataPath)) @@ -2374,20 +2356,39 @@ Result<> DREAM3D::FinishImportingObject(DataStructure& importStructure, DataStru Result DREAM3D::ReadFile(const nx::core::HDF5::FileIO& fileReader, bool preflight) { - // Pipeline pipeline; auto pipeline = ImportPipelineFromFile(fileReader); if(pipeline.invalid()) { return {{nonstd::make_unexpected(std::move(pipeline.errors()))}, std::move(pipeline.warnings())}; } - auto dataStructure = ImportDataStructureFromFile(fileReader, preflight); - if(pipeline.invalid()) + // Preflight import to build placeholder DataStructure and collect top-level paths. + auto preflightDataStructureResult = ImportDataStructureFromFile(fileReader, true); + if(preflightDataStructureResult.invalid()) + { + return {{nonstd::make_unexpected(std::move(preflightDataStructureResult.errors()))}, std::move(preflightDataStructureResult.warnings())}; + } + + if(preflight) + { + return {DREAM3D::FileData{std::move(pipeline.value()), std::move(preflightDataStructureResult.value())}}; + } + + // Collect all DataPaths (ancestors first) from the preflight structure, + // matching the same path set that ReadDREAM3DFilter passes. + std::vector allDataPaths = preflightDataStructureResult.value().getAllDataPaths(); + + // Route through ImportH5ObjectPathsAction so the data store import handler + // (OOC backfill, recovery reattachment) fires if registered. + DataStructure dataStructure; + ImportH5ObjectPathsAction importAction(fileReader.getFilePath(), allDataPaths); + auto importResult = importAction.apply(dataStructure, IDataAction::Mode::Execute); + if(importResult.invalid()) { - return {{nonstd::make_unexpected(std::move(dataStructure.errors()))}, std::move(dataStructure.warnings())}; + return {{nonstd::make_unexpected(std::move(importResult.errors()))}, std::move(importResult.warnings())}; } - return {DREAM3D::FileData{std::move(pipeline.value()), std::move(dataStructure.value())}}; + return {DREAM3D::FileData{std::move(pipeline.value()), std::move(dataStructure)}}; } Result DREAM3D::ReadFile(const std::filesystem::path& path) diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp index fcd5a5daf9..8a41763c5e 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp @@ -146,10 +146,6 @@ SIMPLNX_EXPORT Result<> AppendFile(const std::filesystem::path& path, const Data */ SIMPLNX_EXPORT Result ImportDataStructureFromFile(const nx::core::HDF5::FileIO& fileReader, bool preflight); -SIMPLNX_EXPORT Result> ImportDataObjectFromFile(const nx::core::HDF5::FileIO& fileReader, const DataPath& dataPath); - -SIMPLNX_EXPORT Result>> ImportSelectDataObjectsFromFile(const nx::core::HDF5::FileIO& fileReader, const std::vector& dataPaths); - SIMPLNX_EXPORT Result<> FinishImportingObject(DataStructure& importStructure, DataStructure& dataStructure, const DataPath& dataPath, const nx::core::HDF5::FileIO& fileReader, bool preflight); /** diff --git a/test/DataIOCollectionHooksTest.cpp b/test/DataIOCollectionHooksTest.cpp index 0565758fba..e6d1703cb5 100644 --- a/test/DataIOCollectionHooksTest.cpp +++ b/test/DataIOCollectionHooksTest.cpp @@ -44,8 +44,8 @@ TEST_CASE("DataIOCollectionHooks: format resolver can be unset") REQUIRE(collection.resolveFormat(ds, dp, DataType::float32, 40).empty()); } -TEST_CASE("DataIOCollectionHooks: backfill handler default behavior") +TEST_CASE("DataIOCollectionHooks: data store import handler default behavior") { DataIOCollection collection; - REQUIRE(collection.hasBackfillHandler() == false); + REQUIRE(collection.hasDataStoreImportHandler() == false); } From 09be9201b6c399d0991a97359c046333f805e2b2 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Thu, 9 Apr 2026 21:52:42 -0400 Subject: [PATCH 05/13] REFACTOR: Replace Dream3dIO public API with new LoadDataStructure family MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the old Dream3dIO public API (ReadFile, ImportDataStructureFromFile, FinishImportingObject) with four new purpose-specific functions: - LoadDataStructure(path) — full load with OOC handler support - LoadDataStructureArrays(path, dataPaths) — selective array load with pruning - LoadDataStructureMetadata(path) — metadata-only skeleton (preflight) - LoadDataStructureArraysMetadata(path, dataPaths) — pruned metadata skeleton The new API eliminates the bool preflight parameter in favor of distinct functions, decouples pipeline loading from DataStructure loading, and centralizes the OOC handler integration in a single internal LoadDataStructureWithHandler function. Key changes: DataIOCollection: Add EagerLoadFnc typedef and pass it through the DataStoreImportHandlerFnc signature, replacing the importStructure parameter. The handler can now eager-load individual arrays via callback without knowing Dream3dIO internals. ImportH5ObjectPathsAction: Rewrite to use the new API — preflight calls LoadDataStructureMetadata, execute calls LoadDataStructure. The action no longer manages HDF5 file handles or deferred loading directly; it merges source objects into the pipeline DataStructure via shallow copy. ReadDREAM3DFilter: Switch preflight from ImportDataStructureFromFile(reader, true) to LoadDataStructureMetadata(path), removing manual HDF5 file open. Dream3dIO internals: Move LoadDataObjectFromHDF5, EagerLoadDataFromHDF5, PruneDataStructure, and LoadDataStructureWithHandler into an anonymous namespace. LoadDataStructureWithHandler implements the shared logic: build metadata skeleton, optionally delegate to the OOC import handler, fall back to eager in-core loading. Test callers: Switch ComputeIPFColorsTest, RotateSampleRefFrameTest, DREAM3DFileTest, and H5Test to UnitTest::LoadDataStructure. Add Dream3dLoadingApiTest with coverage for all four new functions. UnitTestCommon: Simplify LoadDataStructure/LoadDataStructureMetadata helpers to delegate directly to the new DREAM3D:: functions. --- .../test/ComputeIPFColorsTest.cpp | 4 +- .../SimplnxCore/Filters/ReadDREAM3DFilter.cpp | 10 +- .../SimplnxCore/test/DREAM3DFileTest.cpp | 9 +- .../test/RotateSampleRefFrameTest.cpp | 6 +- .../IO/Generic/DataIOCollection.cpp | 4 +- .../IO/Generic/DataIOCollection.hpp | 47 ++- .../Actions/ImportH5ObjectPathsAction.cpp | 92 ++-- .../Utilities/Parsing/DREAM3D/Dream3dIO.cpp | 383 +++++++++++------ .../Utilities/Parsing/DREAM3D/Dream3dIO.hpp | 81 ++-- test/CMakeLists.txt | 1 + test/Dream3dLoadingApiTest.cpp | 397 ++++++++++++++++++ test/H5Test.cpp | 11 +- .../simplnx/UnitTest/UnitTestCommon.cpp | 40 +- 13 files changed, 804 insertions(+), 281 deletions(-) create mode 100644 test/Dream3dLoadingApiTest.cpp diff --git a/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp b/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp index 444828ec87..3c21e43f1a 100644 --- a/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp @@ -75,9 +75,7 @@ TEST_CASE("OrientationAnalysis::ComputeIPFColors", "[OrientationAnalysis][Comput // This test file was produced by SIMPL/DREAM3D. our results should match theirs auto exemplarFilePath = fs::path(fmt::format("{}/so3_cubic_high_ipf_001.dream3d", unit_test::k_TestFilesDir)); REQUIRE(fs::exists(exemplarFilePath)); - auto result = DREAM3D::ImportDataStructureFromFile(exemplarFilePath, false); - REQUIRE(result.valid()); - dataStructure = result.value(); + dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); } // Instantiate the filter, a DataStructure object and an Arguments Object diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadDREAM3DFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadDREAM3DFilter.cpp index 664698b6aa..2017736cb3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadDREAM3DFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ReadDREAM3DFilter.cpp @@ -6,7 +6,6 @@ #include "simplnx/Parameters/StringParameter.hpp" #include "simplnx/Pipeline/Pipeline.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" -#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" #include "simplnx/Utilities/SIMPLConversion.hpp" @@ -16,7 +15,6 @@ namespace { constexpr nx::core::int32 k_NoImportPathError = -1; -constexpr nx::core::int32 k_FailedOpenFileIOError = -25; constexpr nx::core::int32 k_UnsupportedPathImportPolicyError = -51; } // namespace @@ -82,16 +80,10 @@ IFilter::PreflightResult ReadDREAM3DFilter::preflightImpl(const DataStructure& d { return {MakeErrorResult(k_NoImportPathError, "Import file path not provided.")}; } - auto fileReader = nx::core::HDF5::FileIO::ReadFile(importData.FilePath); - if(!fileReader.isValid()) - { - return {MakeErrorResult(k_FailedOpenFileIOError, fmt::format("Failed to open the HDF5 file at the specified path: '{}'", importData.FilePath.string()))}; - } - Result result; OutputActions& actions = result.value(); - Result dataStructureResult = DREAM3D::ImportDataStructureFromFile(fileReader, true); + Result dataStructureResult = DREAM3D::LoadDataStructureMetadata(importData.FilePath); if(dataStructureResult.invalid()) { return {ConvertResultTo(ConvertResult(std::move(dataStructureResult)), {})}; diff --git a/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp b/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp index 8091e36d58..367fa4aedb 100644 --- a/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp +++ b/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp @@ -282,10 +282,13 @@ TEST_CASE("DREAM3DFileTest:DREAM3D File IO Test", "[WriteDREAM3DFilter]") // Read .dream3d file { auto fileReader = HDF5::FileIO::ReadFile(GetIODataPath()); - auto fileResult = DREAM3D::ReadFile(fileReader); - SIMPLNX_RESULT_REQUIRE_VALID(fileResult); + auto pipelineResult = DREAM3D::ImportPipelineFromFile(fileReader); + SIMPLNX_RESULT_REQUIRE_VALID(pipelineResult); + auto pipeline = std::move(pipelineResult.value()); - auto [pipeline, dataStructure] = fileResult.value(); + auto dsResult = DREAM3D::LoadDataStructure(GetIODataPath()); + SIMPLNX_RESULT_REQUIRE_VALID(dsResult); + DataStructure dataStructure = std::move(dsResult.value()); // Test reading the DataStructure REQUIRE(dataStructure.getData(DataPath({DataNames::k_Group1Name})) != nullptr); diff --git a/src/Plugins/SimplnxCore/test/RotateSampleRefFrameTest.cpp b/src/Plugins/SimplnxCore/test/RotateSampleRefFrameTest.cpp index e65d150cc4..340ce9db5f 100644 --- a/src/Plugins/SimplnxCore/test/RotateSampleRefFrameTest.cpp +++ b/src/Plugins/SimplnxCore/test/RotateSampleRefFrameTest.cpp @@ -70,11 +70,7 @@ TEST_CASE("SimplnxCore::RotateSampleRefFrame", "[Core][RotateSampleRefFrameFilte const DataPath k_OriginalGeomPath({"Original"}); - Result dataStructureResult = - DREAM3D::ImportDataStructureFromFile(fs::path(fmt::format("{}/Rotate_Sample_Ref_Frame_Test_v3/Rotate_Sample_Ref_Frame_Test_v3.dream3d", nx::core::unit_test::k_TestFilesDir)), false); - SIMPLNX_RESULT_REQUIRE_VALID(dataStructureResult); - - DataStructure dataStructure = std::move(dataStructureResult.value()); + DataStructure dataStructure = UnitTest::LoadDataStructure(fs::path(fmt::format("{}/Rotate_Sample_Ref_Frame_Test_v3/Rotate_Sample_Ref_Frame_Test_v3.dream3d", nx::core::unit_test::k_TestFilesDir))); const auto* originalImageGeom = dataStructure.getDataAs(k_OriginalGeomPath); REQUIRE(originalImageGeom != nullptr); diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index 17d9a4fa0a..2513318b7b 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -249,7 +249,7 @@ bool DataIOCollection::hasDataStoreImportHandler() const return static_cast(m_DataStoreImportHandler); } -Result<> DataIOCollection::runDataStoreImportHandler(DataStructure& importStructure, DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader) +Result<> DataIOCollection::runDataStoreImportHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader, EagerLoadFnc eagerLoad) { // If no handler is installed, this is the in-memory code path: placeholders // will be replaced elsewhere (or eagerly loaded). Return success. @@ -259,7 +259,7 @@ Result<> DataIOCollection::runDataStoreImportHandler(DataStructure& importStruct } // Delegate to the plugin-provided handler, which loads all data stores — // in-core, OOC, or recovered — for the imported paths. - return m_DataStoreImportHandler(importStructure, dataStructure, paths, fileReader); + return m_DataStoreImportHandler(dataStructure, paths, fileReader, std::move(eagerLoad)); } std::vector DataIOCollection::getFormatNames() const diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index c126e09e2e..d55326cea6 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -66,6 +66,13 @@ class SIMPLNX_EXPORT DataIOCollection */ using FormatResolverFnc = std::function; + /** + * @brief Callback that eagerly loads a single DataObject's data from HDF5 into memory. + * Constructed by the loading infrastructure and passed to the data store import handler. + * The handler calls this for arrays the format resolver says should be in-core. + */ + using EagerLoadFnc = std::function(DataStructure& dataStructure, const DataPath& path)>; + /** * @brief Callback responsible for ALL data store loading from .dream3d files — * in-core, OOC, and recovery. @@ -73,23 +80,29 @@ class SIMPLNX_EXPORT DataIOCollection * During import, the HDF5 reader creates lightweight placeholder stores * (EmptyDataStore, EmptyListStore, EmptyStringStore) for every array so that * the DataStructure's topology is complete without loading data. The data - * store import handler is then called with the list of imported DataPaths and - * the still-open HDF5 file reader, giving the OOC plugin the opportunity to - * replace each placeholder with a real store — either a read-only OOC - * reference that lazily reads chunks from the same file, an eagerly-loaded - * in-core store, or a recovered store from a prior session. + * store import handler is then called with the list of imported DataPaths, + * the still-open HDF5 file reader, and an eager-load callback, giving the OOC + * plugin the opportunity to replace each placeholder with a real store — either + * a read-only OOC reference that lazily reads chunks from the same file, an + * eagerly-loaded in-core store (by invoking @p eagerLoad), or a recovered store + * from a prior session. + * + * The @p eagerLoad callback encapsulates the Dream3dIO loading infrastructure so + * the handler does not need to know about file format versions or internal reader + * details — it simply calls eagerLoad(dataStructure, path) for any array that + * should be brought fully into memory. * * Only one data store import handler may be registered at a time. If no * handler is registered, the placeholders remain (which is valid for * in-memory workflows that eagerly load data through a different path). * - * @param importStructure The intermediate DataStructure from the HDF5 reader (holds reader factories) - * @param dataStructure The target DataStructure containing placeholder stores to replace - * @param paths DataPaths of the objects imported from the file - * @param fileReader Open HDF5 file reader for the source .dream3d file + * @param dataStructure The target DataStructure containing placeholder stores to replace + * @param paths DataPaths of the objects imported from the file + * @param fileReader Open HDF5 file reader for the source .dream3d file + * @param eagerLoad Callback that loads a single array's data from HDF5 into memory * @return Result<> indicating success or describing any errors */ - using DataStoreImportHandlerFnc = std::function(DataStructure& importStructure, DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader)>; + using DataStoreImportHandlerFnc = std::function(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader, EagerLoadFnc eagerLoad)>; /** * @brief Callback that can intercept and override how a DataObject is written @@ -185,13 +198,17 @@ class SIMPLNX_EXPORT DataIOCollection * Result. This allows the in-memory code path to skip store import entirely * while the OOC code path processes every imported array. * - * @param importStructure The intermediate DataStructure from the HDF5 reader (holds reader factories) - * @param dataStructure The target DataStructure whose placeholder stores should be replaced - * @param paths DataPaths of the objects imported from the file - * @param fileReader Open HDF5 file reader for the source .dream3d file + * The @p eagerLoad callback is forwarded directly to the handler. It wraps the + * Dream3dIO loading infrastructure so the handler can bring individual arrays + * fully into memory without depending on Dream3dIO internals. + * + * @param dataStructure The target DataStructure whose placeholder stores should be replaced + * @param paths DataPaths of the objects imported from the file + * @param fileReader Open HDF5 file reader for the source .dream3d file + * @param eagerLoad Callback that loads a single array's data from HDF5 into memory * @return Result<> indicating success, or containing errors from the handler */ - Result<> runDataStoreImportHandler(DataStructure& importStructure, DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader); + Result<> runDataStoreImportHandler(DataStructure& dataStructure, const std::vector& paths, const nx::core::HDF5::FileIO& fileReader, EagerLoadFnc eagerLoad); /** * @brief Registers or clears the write-array-override callback. diff --git a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp index fe81e92115..28e28aeb0b 100644 --- a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp +++ b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp @@ -1,24 +1,16 @@ #include "ImportH5ObjectPathsAction.hpp" -#include "simplnx/Utilities/DataStoreUtilities.hpp" +#include "simplnx/Common/StringLiteralFormatting.hpp" +#include "simplnx/DataStructure/BaseGroup.hpp" +#include "simplnx/DataStructure/DataObject.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" -#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" #include #include -#include using namespace nx::core; -namespace -{ -void sortImportPaths(std::vector& importPaths) -{ - std::sort(importPaths.begin(), importPaths.end(), [](const DataPath& first, const DataPath& second) { return first.getLength() < second.getLength(); }); -} -} // namespace - namespace nx::core { ImportH5ObjectPathsAction::ImportH5ObjectPathsAction(const std::filesystem::path& importFile, const PathsType& paths) @@ -26,7 +18,7 @@ ImportH5ObjectPathsAction::ImportH5ObjectPathsAction(const std::filesystem::path , m_H5FilePath(importFile) , m_Paths(paths) { - sortImportPaths(m_Paths); + std::sort(m_Paths.begin(), m_Paths.end(), [](const DataPath& a, const DataPath& b) { return a.getLength() < b.getLength(); }); } ImportH5ObjectPathsAction::~ImportH5ObjectPathsAction() noexcept = default; @@ -35,64 +27,58 @@ Result<> ImportH5ObjectPathsAction::apply(DataStructure& dataStructure, Mode mod { static constexpr StringLiteral prefix = "ImportH5ObjectPathsAction: "; - auto fileReader = nx::core::HDF5::FileIO::ReadFile(m_H5FilePath); - // Import as a preflight data structure to start to conserve memory and only allocate the data you want later - Result dataStructureResult = DREAM3D::ImportDataStructureFromFile(fileReader, true); - if(dataStructureResult.invalid()) + // Get the source DataStructure — metadata only for preflight, loaded arrays for execute + // Preflight: metadata only. Execute: full load (not LoadDataStructureArrays, because the + // merge loop below selectively copies only m_Paths — pruning would break Geometry/AttributeMatrix + // relationships in the source structure before the merge has a chance to pick the right objects). + auto result = (mode == Mode::Preflight) ? DREAM3D::LoadDataStructureMetadata(m_H5FilePath) : DREAM3D::LoadDataStructure(m_H5FilePath); + + if(result.invalid()) { - return ConvertResult(std::move(dataStructureResult)); + return ConvertResult(std::move(result)); } - // Ensure there are no conflicting DataObject ID values - DataStructure importStructure = std::move(dataStructureResult.value()); - importStructure.resetIds(dataStructure.getNextId()); - - const bool preflighting = mode == Mode::Preflight; + DataStructure sourceStructure = std::move(result.value()); + sourceStructure.resetIds(dataStructure.getNextId()); - // Generic: "does a plugin want to defer loading and take over finalization?" - bool useDeferredLoad = false; - if(!preflighting) - { - auto ioCollection = DataStoreUtilities::GetIOCollection(); - useDeferredLoad = ioCollection->hasDataStoreImportHandler(); - } + // Merge source objects into the pipeline's DataStructure. + // Sort paths shortest-first so parents are inserted before children. + auto sortedPaths = m_Paths; + std::sort(sortedPaths.begin(), sortedPaths.end(), [](const DataPath& a, const DataPath& b) { return a.getLength() < b.getLength(); }); - std::stringstream errorMessages; - for(const auto& targetPath : m_Paths) + for(const auto& targetPath : sortedPaths) { if(dataStructure.getDataAs(targetPath) != nullptr) { - return MakeErrorResult(-6203, fmt::format("{}Unable to import DataObject at '{}' because an object already exists there. Consider a rename of existing object.", prefix, targetPath.toString())); + return MakeErrorResult(-6203, fmt::format("{}Unable to import DataObject at '{}' because an object " + "already exists at that path. Consider renaming the existing object before importing, or " + "exclude this path from the import selection.", + prefix, targetPath.toString())); } - // When deferring loading, pass preflight=true to avoid the expensive - // data copy. The resulting placeholder stores are finalized by the - // registered data store import handler below. - auto result = DREAM3D::FinishImportingObject(importStructure, dataStructure, targetPath, fileReader, preflighting || useDeferredLoad); - if(result.invalid()) + if(!sourceStructure.containsData(targetPath)) { - for(const auto& errorResult : result.errors()) - { - errorMessages << errorResult.message << std::endl; - } + continue; } - } - if(!errorMessages.str().empty()) - { - return MakeErrorResult(-6201, errorMessages.str()); - } - if(useDeferredLoad) - { - auto ioCollection = DataStoreUtilities::GetIOCollection(); - auto importResult = ioCollection->runDataStoreImportHandler(importStructure, dataStructure, m_Paths, fileReader); - if(importResult.invalid()) + // Shallow-copy the object from the source structure (which has real stores) + // and insert it into the pipeline's DataStructure. Clear children on groups + // because child objects will be inserted by their own paths in the loop. + const auto sourceObject = sourceStructure.getSharedData(targetPath); + const auto objectCopy = std::shared_ptr(sourceObject->shallowCopy()); + if(const auto group = std::dynamic_pointer_cast(objectCopy); group != nullptr) + { + group->clear(); + } + if(!dataStructure.insert(objectCopy, targetPath.getParent())) { - return importResult; + return MakeErrorResult(-6202, fmt::format("{}Unable to insert DataObject at path '{}' into the DataStructure. " + "The parent path '{}' may not exist.", + prefix, targetPath.toString(), targetPath.getParent().toString())); } } - return ConvertResult(std::move(dataStructureResult)); + return {}; } IDataAction::UniquePointer ImportH5ObjectPathsAction::clone() const diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp index 5dbec18d06..05d445ab31 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp @@ -2,6 +2,7 @@ #include "simplnx/Common/Aliases.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/BaseGroup.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataGroup.hpp" #include "simplnx/DataStructure/DataStore.hpp" @@ -23,7 +24,6 @@ #include "simplnx/DataStructure/NeighborList.hpp" #include "simplnx/DataStructure/StringArray.hpp" #include "simplnx/DataStructure/StringStore.hpp" -#include "simplnx/Filter/Actions/ImportH5ObjectPathsAction.hpp" #include "simplnx/Pipeline/Pipeline.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -2221,33 +2221,6 @@ Result ImportLegacyDataStructure(const nx::core::HDF5::FileIO& fi return nx::core::ConvertResultTo(std::move(result), std::move(dataStructure)); } -Result DREAM3D::ImportDataStructureFromFile(const nx::core::HDF5::FileIO& fileReader, bool preflight) -{ - const auto fileVersion = GetFileVersion(fileReader); - if(fileVersion == k_CurrentFileVersion) - { - return ImportDataStructureV8(fileReader, preflight); - } - else if(fileVersion == k_LegacyFileVersion) - { - return ImportLegacyDataStructure(fileReader, preflight); - } - // Unsupported file version - return MakeErrorResult(k_InvalidDataStructureVersion, fmt::format("Could not parse DataStructure version {}. Expected versions: {} or {}. Actual value: {}", fileVersion, - k_CurrentFileVersion, k_LegacyFileVersion, fileVersion)); -} - -Result DREAM3D::ImportDataStructureFromFile(const std::filesystem::path& filePath, bool preflight) -{ - auto fileReader = nx::core::HDF5::FileIO::ReadFile(filePath); - if(!fileReader.isValid()) - { - return MakeErrorResult(-1, fmt::format("DREAM3D::ImportDataStructureFromFile: Unable to open '{}' for reading", filePath.string())); - } - - return ImportDataStructureFromFile(fileReader, preflight); -} - Result DREAM3D::ImportPipelineFromFile(const nx::core::HDF5::FileIO& fileReader) { Result pipelineJson = ImportPipelineJsonFromFile(fileReader); @@ -2314,96 +2287,6 @@ Result DREAM3D::ImportPipelineJsonFromFile(const std::filesystem return ImportPipelineJsonFromFile(fileReader); } -Result<> DREAM3D::FinishImportingObject(DataStructure& importStructure, DataStructure& dataStructure, const DataPath& dataPath, const nx::core::HDF5::FileIO& fileReader, bool preflight) -{ - if(!importStructure.containsData(dataPath)) - { - return MakeErrorResult(-6200, fmt::format("DataStructure Object Path '{}' does not exist for importing.", dataPath.toString())); - } - const auto importObject = importStructure.getSharedData(dataPath); - const auto importData = std::shared_ptr(importObject->shallowCopy()); - // Clear all children before inserting into the DataStructure - if(const auto importGroup = std::dynamic_pointer_cast(importData); importGroup != nullptr) - { - importGroup->clear(); - } - - if(!dataStructure.insert(importData, dataPath.getParent())) - { - return MakeErrorResult(-6202, fmt::format("Unable to insert DataObject at DatPath '{}' into the DataStructure", dataPath.toString())); - } - if(!preflight) - { - const auto dataPtr = dataStructure.getSharedData(dataPath); - if(dataPtr == nullptr) - { - return MakeErrorResult(-1502234, fmt::format("Cannot finish importing HDF5 data at DataPath '{}'. DataObject does not exist to copy data into.", dataPath.toString())); - } - - const auto fileVersion = GetFileVersion(fileReader); - if(fileVersion == k_CurrentFileVersion) - { - return HDF5::DataStructureReader::FinishImportingObject(dataStructure, fileReader, dataPath); - } - else if(fileVersion == k_LegacyFileVersion) - { - const auto dataStructureReader = fileReader.openGroup(k_LegacyDataStructureGroupTag); - return FinishImportingLegacyDataObject(dataStructure, dataStructureReader, dataPath); - } - } - return {}; -} - -Result DREAM3D::ReadFile(const nx::core::HDF5::FileIO& fileReader, bool preflight) -{ - auto pipeline = ImportPipelineFromFile(fileReader); - if(pipeline.invalid()) - { - return {{nonstd::make_unexpected(std::move(pipeline.errors()))}, std::move(pipeline.warnings())}; - } - - // Preflight import to build placeholder DataStructure and collect top-level paths. - auto preflightDataStructureResult = ImportDataStructureFromFile(fileReader, true); - if(preflightDataStructureResult.invalid()) - { - return {{nonstd::make_unexpected(std::move(preflightDataStructureResult.errors()))}, std::move(preflightDataStructureResult.warnings())}; - } - - if(preflight) - { - return {DREAM3D::FileData{std::move(pipeline.value()), std::move(preflightDataStructureResult.value())}}; - } - - // Collect all DataPaths (ancestors first) from the preflight structure, - // matching the same path set that ReadDREAM3DFilter passes. - std::vector allDataPaths = preflightDataStructureResult.value().getAllDataPaths(); - - // Route through ImportH5ObjectPathsAction so the data store import handler - // (OOC backfill, recovery reattachment) fires if registered. - DataStructure dataStructure; - ImportH5ObjectPathsAction importAction(fileReader.getFilePath(), allDataPaths); - auto importResult = importAction.apply(dataStructure, IDataAction::Mode::Execute); - if(importResult.invalid()) - { - return {{nonstd::make_unexpected(std::move(importResult.errors()))}, std::move(importResult.warnings())}; - } - - return {DREAM3D::FileData{std::move(pipeline.value()), std::move(dataStructure)}}; -} - -Result DREAM3D::ReadFile(const std::filesystem::path& path) -{ - auto reader = nx::core::HDF5::FileIO::ReadFile(path); - nx::core::HDF5::ErrorType error = 0; - - Result fileData = ReadFile(reader, error); - if(error < 0) - { - return MakeErrorResult(-1, fmt::format("DREAM3D::ReadFile: Unable to read '{}'", path.string())); - } - return fileData; -} - Result<> WritePipeline(nx::core::HDF5::FileIO& fileWriter, const Pipeline& pipeline) { if(!fileWriter.isValid()) @@ -2563,3 +2446,267 @@ std::vector DREAM3D::ExpandSelectedPathsToDescendants(const return expandedDataPaths; } + +namespace +{ +// --------------------------------------------------------------------------- +// Internal helpers for the new LoadDataStructure* public API +// --------------------------------------------------------------------------- + +/** + * @brief Builds a metadata-only (preflight) DataStructure from an open HDF5 file. + * This is a copy of DREAM3D::ImportDataStructureFromFile with preflight hardcoded to true. + */ +Result LoadDataStructureMetadataInternal(const nx::core::HDF5::FileIO& fileReader) +{ + const auto fileVersion = DREAM3D::GetFileVersion(fileReader); + if(fileVersion == DREAM3D::k_CurrentFileVersion) + { + return ImportDataStructureV8(fileReader, true); + } + else if(fileVersion == DREAM3D::k_LegacyFileVersion) + { + return ImportLegacyDataStructure(fileReader, true); + } + // Unsupported file version + return MakeErrorResult(DREAM3D::k_InvalidDataStructureVersion, fmt::format("Could not parse DataStructure version {}. Expected versions: {} or {}. Actual value: {}", fileVersion, + DREAM3D::k_CurrentFileVersion, DREAM3D::k_LegacyFileVersion, fileVersion)); +} + +/** + * @brief Loads a single DataObject from HDF5 into the target DataStructure. + * This is a copy of DREAM3D::FinishImportingObject placed in the anonymous namespace. + */ +Result<> LoadDataObjectFromHDF5(DataStructure& importStructure, DataStructure& dataStructure, const DataPath& dataPath, const nx::core::HDF5::FileIO& fileReader, bool preflight) +{ + if(!importStructure.containsData(dataPath)) + { + return MakeErrorResult(-6200, fmt::format("DataStructure Object Path '{}' does not exist for importing.", dataPath.toString())); + } + const auto importObject = importStructure.getSharedData(dataPath); + const auto importData = std::shared_ptr(importObject->shallowCopy()); + // Clear all children before inserting into the DataStructure + if(const auto importGroup = std::dynamic_pointer_cast(importData); importGroup != nullptr) + { + importGroup->clear(); + } + + if(!dataStructure.insert(importData, dataPath.getParent())) + { + return MakeErrorResult(-6202, fmt::format("Unable to insert DataObject at DataPath '{}' into the DataStructure", dataPath.toString())); + } + if(!preflight) + { + const auto dataPtr = dataStructure.getSharedData(dataPath); + if(dataPtr == nullptr) + { + return MakeErrorResult(-1502234, fmt::format("Cannot finish importing HDF5 data at DataPath '{}'. DataObject does not exist to copy data into.", dataPath.toString())); + } + + const auto fileVersion = DREAM3D::GetFileVersion(fileReader); + if(fileVersion == DREAM3D::k_CurrentFileVersion) + { + return HDF5::DataStructureReader::FinishImportingObject(dataStructure, fileReader, dataPath); + } + else if(fileVersion == DREAM3D::k_LegacyFileVersion) + { + const auto dataStructureReader = fileReader.openGroup(k_LegacyDataStructureGroupTag); + return FinishImportingLegacyDataObject(dataStructure, dataStructureReader, dataPath); + } + } + return {}; +} + +/** + * @brief Loads data from HDF5 into an already-inserted DataObject. + * + * Unlike LoadDataObjectFromHDF5, this does NOT insert the object — it only + * reads the HDF5 data for an object that is already present in dataStructure. + * Used as the eagerLoad callback when the OOC handler decides an array should + * be loaded in-core (below the size threshold). + */ +Result<> EagerLoadDataFromHDF5(DataStructure& dataStructure, const DataPath& dataPath, const nx::core::HDF5::FileIO& fileReader) +{ + const auto dataPtr = dataStructure.getSharedData(dataPath); + if(dataPtr == nullptr) + { + return MakeErrorResult(-6203, fmt::format("Cannot eager-load HDF5 data at DataPath '{}'. DataObject does not exist in the DataStructure.", dataPath.toString())); + } + + const auto fileVersion = DREAM3D::GetFileVersion(fileReader); + if(fileVersion == DREAM3D::k_CurrentFileVersion) + { + return HDF5::DataStructureReader::FinishImportingObject(dataStructure, fileReader, dataPath); + } + else if(fileVersion == DREAM3D::k_LegacyFileVersion) + { + const auto dataStructureReader = fileReader.openGroup(k_LegacyDataStructureGroupTag); + return FinishImportingLegacyDataObject(dataStructure, dataStructureReader, dataPath); + } + return {}; +} + +/** + * @brief Removes all DataObjects from ds that are not an ancestor of or equal to any path in keepPaths. + */ +void PruneDataStructure(DataStructure& ds, const std::vector& keepPaths) +{ + auto allPaths = ds.getAllDataPaths(); + // Sort longest-first so children are removed before parents + std::sort(allPaths.begin(), allPaths.end(), [](const DataPath& a, const DataPath& b) { return a.getLength() > b.getLength(); }); + + for(const auto& existingPath : allPaths) + { + bool isNeeded = false; + for(const auto& requestedPath : keepPaths) + { + // Keep if it equals a requested path or is an ancestor of one + if(existingPath == requestedPath) + { + isNeeded = true; + break; + } + // Check if existingPath is an ancestor of requestedPath + const auto& existingVec = existingPath.getPathVector(); + const auto& requestedVec = requestedPath.getPathVector(); + if(existingVec.size() < requestedVec.size()) + { + bool isPrefix = true; + for(usize i = 0; i < existingVec.size(); ++i) + { + if(existingVec[i] != requestedVec[i]) + { + isPrefix = false; + break; + } + } + if(isPrefix) + { + isNeeded = true; + break; + } + } + } + if(!isNeeded) + { + ds.removeData(existingPath); + } + } +} + +/** + * @brief Shared logic for LoadDataStructure and LoadDataStructureArrays. + * Builds a metadata skeleton, then either delegates to the registered data store import handler + * (e.g. OOC) or eager-loads everything in-core. + * + * Follows the same pattern as ImportH5ObjectPathsAction::apply: + * 1. Preflight-import to get the metadata skeleton (importStructure) + * 2. Expand paths to include ancestors, sorted shortest-first + * 3. Insert each object via LoadDataObjectFromHDF5 (shallow copy + insert + optional data load) + * 4. If a handler is registered, run it for deferred loading + */ +Result LoadDataStructureWithHandler(const std::filesystem::path& filePath, const std::vector& paths) +{ + auto fileReader = nx::core::HDF5::FileIO::ReadFile(filePath); + if(!fileReader.isValid()) + { + return MakeErrorResult(-1, fmt::format("Failed to open .dream3d file '{}'. Check that the file exists and is a valid HDF5 file.", filePath.string())); + } + + // Build placeholder skeleton + auto metadataResult = LoadDataStructureMetadataInternal(fileReader); + if(metadataResult.invalid()) + { + return metadataResult; + } + DataStructure importStructure = std::move(metadataResult.value()); + + // Reopen file for data reading + auto dataFileReader = nx::core::HDF5::FileIO::ReadFile(filePath); + + // Check if a handler is registered (e.g. OOC plugin) + auto ioCollection = DataStoreUtilities::GetIOCollection(); + const bool useDeferredLoad = ioCollection->hasDataStoreImportHandler(); + + // Expand to include ancestor containers, sorted shortest-first + auto allPaths = DREAM3D::ExpandSelectedPathsToAncestors(paths); + std::sort(allPaths.begin(), allPaths.end(), [](const DataPath& a, const DataPath& b) { return a.getLength() < b.getLength(); }); + + // Insert each object into the target DataStructure. + // When deferring, pass preflight=true to insert placeholders without loading data. + // When not deferring, pass preflight=false to insert and load data immediately. + DataStructure dataStructure; + for(const auto& objectPath : allPaths) + { + auto result = LoadDataObjectFromHDF5(importStructure, dataStructure, objectPath, dataFileReader, useDeferredLoad); + if(result.invalid()) + { + return ConvertInvalidResult(std::move(result)); + } + } + + // If a handler is registered, let it finalize loading (e.g. attach OOC stores) + std::vector handlerWarnings; + if(useDeferredLoad) + { + auto eagerLoad = [&dataFileReader](DataStructure& ds, const DataPath& path) -> Result<> { return EagerLoadDataFromHDF5(ds, path, dataFileReader); }; + auto handlerResult = ioCollection->runDataStoreImportHandler(dataStructure, paths, dataFileReader, eagerLoad); + if(handlerResult.invalid()) + { + return ConvertInvalidResult(std::move(handlerResult)); + } + handlerWarnings = std::move(handlerResult.warnings()); + } + + Result finalResult{std::move(dataStructure)}; + finalResult.warnings() = std::move(handlerWarnings); + return finalResult; +} +} // namespace + +// --------------------------------------------------------------------------- +// New public LoadDataStructure* API +// --------------------------------------------------------------------------- + +Result DREAM3D::LoadDataStructureMetadata(const std::filesystem::path& path) +{ + auto fileReader = nx::core::HDF5::FileIO::ReadFile(path); + if(!fileReader.isValid()) + { + return MakeErrorResult(-1, fmt::format("Failed to open .dream3d file '{}'. Check that the file exists and is a valid HDF5 file.", path.string())); + } + return LoadDataStructureMetadataInternal(fileReader); +} + +Result DREAM3D::LoadDataStructure(const std::filesystem::path& path) +{ + auto metadataResult = DREAM3D::LoadDataStructureMetadata(path); + if(metadataResult.invalid()) + { + return metadataResult; + } + std::vector allPaths = metadataResult.value().getAllDataPaths(); + return LoadDataStructureWithHandler(path, allPaths); +} + +Result DREAM3D::LoadDataStructureArrays(const std::filesystem::path& path, const std::vector& dataPaths) +{ + auto result = LoadDataStructureWithHandler(path, dataPaths); + if(result.invalid()) + { + return result; + } + PruneDataStructure(result.value(), dataPaths); + return result; +} + +Result DREAM3D::LoadDataStructureArraysMetadata(const std::filesystem::path& path, const std::vector& dataPaths) +{ + auto result = DREAM3D::LoadDataStructureMetadata(path); + if(result.invalid()) + { + return result; + } + PruneDataStructure(result.value(), dataPaths); + return result; +} diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp index 8a41763c5e..92b70a8052 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp @@ -52,27 +52,58 @@ SIMPLNX_EXPORT FileVersionType GetFileVersion(const nx::core::HDF5::FileIO& file SIMPLNX_EXPORT PipelineVersionType GetPipelineVersion(const nx::core::HDF5::FileIO& fileReader); /** - * @brief Imports and returns the Pipeline / DataStructure pair from the target - * .dream3d file. + * @brief Loads a complete DataStructure from a .dream3d file with all arrays + * receiving real data stores (in-core or OOC via the registered import handler). * - * This method imports both current and legacy DataStructures but will return - * an empty Pipeline when given a legacy file. - * @param fileReader - * @param preflight = false - * @return FileData + * Supports both v8.0 and legacy v7.0 file formats. When an OOC import handler + * is registered (via DataIOCollection), it decides whether each array becomes + * an in-core DataStore or a lazy OOC store backed by the HDF5 file. + * + * @param path Filesystem path to the .dream3d file + * @return Result containing the fully loaded DataStructure, or errors on failure */ -SIMPLNX_EXPORT Result ReadFile(const nx::core::HDF5::FileIO& fileReader, bool preflight = false); +SIMPLNX_EXPORT Result LoadDataStructure(const std::filesystem::path& path); /** - * @brief Imports and returns the Pipeline / DataStructure pair from the target - * .dream3d file. + * @brief Loads specific arrays from a .dream3d file with real data stores, + * pruning all unrequested objects from the result. * - * This method imports both current and legacy DataStructures but will return - * an empty Pipeline and warning when given a legacy file. - * @param path - * @return Result + * Only the requested arrays (and their ancestor containers) are present in + * the returned DataStructure. No Empty placeholder stores remain — every + * array in the result has been fully loaded or attached to an OOC store. + * + * @param path Filesystem path to the .dream3d file + * @param dataPaths The specific DataPaths to load from the file + * @return Result containing the pruned DataStructure with only requested arrays + */ +SIMPLNX_EXPORT Result LoadDataStructureArrays(const std::filesystem::path& path, const std::vector& dataPaths); + +/** + * @brief Loads the topology (metadata skeleton) of a .dream3d file without + * loading any array data. All DataArrays receive Empty placeholder stores. + * + * This is the preflight/metadata-only path: the returned DataStructure has + * the complete hierarchy (geometries, attribute matrices, arrays) but none + * of the arrays contain real data. + * + * @param path Filesystem path to the .dream3d file + * @return Result containing the metadata-only DataStructure with Empty stores + */ +SIMPLNX_EXPORT Result LoadDataStructureMetadata(const std::filesystem::path& path); + +/** + * @brief Loads the topology (metadata skeleton) for specific arrays from a + * .dream3d file. All arrays receive Empty placeholder stores, and unrequested + * objects are pruned from the result. + * + * Combines the metadata-only behavior of LoadDataStructureMetadata with the + * path-based pruning of LoadDataStructureArrays. + * + * @param path Filesystem path to the .dream3d file + * @param dataPaths The specific DataPaths whose metadata to load + * @return Result containing the pruned metadata-only DataStructure */ -SIMPLNX_EXPORT Result ReadFile(const std::filesystem::path& path); +SIMPLNX_EXPORT Result LoadDataStructureArraysMetadata(const std::filesystem::path& path, const std::vector& dataPaths); /** * @brief Writes a .dream3d file with the specified data. @@ -136,26 +167,6 @@ SIMPLNX_EXPORT Result<> WriteRecoveryFile(const std::filesystem::path& path, con */ SIMPLNX_EXPORT Result<> AppendFile(const std::filesystem::path& path, const DataStructure& dataStructure, const DataPath& dataPath); -/** - * @brief Imports and returns the DataStructure from the target .dream3d file. - * - * This method imports both current and legacy DataStructures. - * @param fileReader - * @param preflight = false - * @return DataStructure - */ -SIMPLNX_EXPORT Result ImportDataStructureFromFile(const nx::core::HDF5::FileIO& fileReader, bool preflight); - -SIMPLNX_EXPORT Result<> FinishImportingObject(DataStructure& importStructure, DataStructure& dataStructure, const DataPath& dataPath, const nx::core::HDF5::FileIO& fileReader, bool preflight); - -/** - * @brief Imports and returns the DataStructure from the target .dream3d file. - * This method imports both current and legacy DataStructures. - * @param filePath - * @return DataStructure - */ -SIMPLNX_EXPORT Result ImportDataStructureFromFile(const std::filesystem::path& filePath, bool preflight); - /** * @brief Imports and returns a Pipeline from the target .dream3d file. * diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 4c668b6804..e178f7fe39 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -27,6 +27,7 @@ add_executable(simplnx_test DataArrayTest.cpp DataIOCollectionHooksTest.cpp DataPathTest.cpp + Dream3dLoadingApiTest.cpp DataStructObserver.hpp DataStructObserver.cpp DataStructTest.cpp diff --git a/test/Dream3dLoadingApiTest.cpp b/test/Dream3dLoadingApiTest.cpp new file mode 100644 index 0000000000..b52226d055 --- /dev/null +++ b/test/Dream3dLoadingApiTest.cpp @@ -0,0 +1,397 @@ +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/DataStore.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/DataStructure/EmptyDataStore.hpp" +#include "simplnx/DataStructure/IDataStore.hpp" +#include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" + +#include "simplnx/unit_test/simplnx_test_dirs.hpp" + +#include + +#include + +namespace fs = std::filesystem; +using namespace nx::core; + +namespace +{ +// --------------------------------------------------------------------------- +// Constants for path construction +// --------------------------------------------------------------------------- +constexpr StringLiteral k_GroupName = "TopGroup"; +constexpr StringLiteral k_SmallAttrMatName = "SmallAM"; +constexpr StringLiteral k_LargeAttrMatName = "LargeAM"; +constexpr StringLiteral k_SmallArrayName = "SmallArray"; +constexpr StringLiteral k_LargeArrayName = "LargeArray"; + +constexpr usize k_SmallArraySize = 10; +constexpr usize k_LargeArraySize = 100; + +// Common DataPaths for the simple test structure +const DataPath k_GroupPath({k_GroupName}); +const DataPath k_SmallAMPath({k_GroupName, k_SmallAttrMatName}); +const DataPath k_LargeAMPath({k_GroupName, k_LargeAttrMatName}); +const DataPath k_SmallArrayPath({k_GroupName, k_SmallAttrMatName, k_SmallArrayName}); +const DataPath k_LargeArrayPath({k_GroupName, k_LargeAttrMatName, k_LargeArrayName}); + +// Paths used in the multi-group prune test (scenario 7) +constexpr StringLiteral k_GroupAName = "GroupA"; +constexpr StringLiteral k_GroupBName = "GroupB"; +constexpr StringLiteral k_AttrMatAName = "AttrMatA"; +constexpr StringLiteral k_AttrMatBName = "AttrMatB"; +constexpr StringLiteral k_ArrayA1Name = "ArrayA1"; +constexpr StringLiteral k_ArrayA2Name = "ArrayA2"; +constexpr StringLiteral k_ArrayB1Name = "ArrayB1"; +constexpr StringLiteral k_ArrayB2Name = "ArrayB2"; +constexpr usize k_PruneArraySize = 20; + +const DataPath k_GroupAPath({k_GroupAName}); +const DataPath k_AttrMatAPath({k_GroupAName, k_AttrMatAName}); +const DataPath k_ArrayA1Path({k_GroupAName, k_AttrMatAName, k_ArrayA1Name}); +const DataPath k_ArrayA2Path({k_GroupAName, k_AttrMatAName, k_ArrayA2Name}); +const DataPath k_GroupBPath({k_GroupBName}); +const DataPath k_AttrMatBPath({k_GroupBName, k_AttrMatBName}); +const DataPath k_ArrayB1Path({k_GroupBName, k_AttrMatBName, k_ArrayB1Name}); +const DataPath k_ArrayB2Path({k_GroupBName, k_AttrMatBName, k_ArrayB2Name}); + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +fs::path GetTestOutputDir() +{ + return fs::path(unit_test::k_BinaryTestOutputDir.view()); +} + +/** + * @brief Creates a DataStructure with the hierarchy: + * TopGroup / SmallAM / SmallArray (10 x int32) + * / LargeAM / LargeArray (100 x float32) + * + * Two separate AttributeMatrices are needed because AM enforces that all + * child arrays share the same tuple dimensions. + * SmallArray values are filled with i * 3, LargeArray with i * 1.5f. + */ +DataStructure CreateSimpleTestDataStructure() +{ + DataStructure ds; + + auto* group = DataGroup::Create(ds, k_GroupName); + REQUIRE(group != nullptr); + + // SmallAM holds SmallArray (10 tuples) + auto* smallAM = AttributeMatrix::Create(ds, k_SmallAttrMatName, {k_SmallArraySize}, group->getId()); + REQUIRE(smallAM != nullptr); + + auto smallStore = std::make_unique>(std::vector{k_SmallArraySize}, std::vector{1}, static_cast(0)); + for(usize i = 0; i < k_SmallArraySize; ++i) + { + smallStore->setValue(i, static_cast(i * 3)); + } + auto* smallArray = DataArray::Create(ds, k_SmallArrayName, std::move(smallStore), smallAM->getId()); + REQUIRE(smallArray != nullptr); + + // LargeAM holds LargeArray (100 tuples) + auto* largeAM = AttributeMatrix::Create(ds, k_LargeAttrMatName, {k_LargeArraySize}, group->getId()); + REQUIRE(largeAM != nullptr); + + auto largeStore = std::make_unique>(std::vector{k_LargeArraySize}, std::vector{1}, static_cast(0)); + for(usize i = 0; i < k_LargeArraySize; ++i) + { + largeStore->setValue(i, static_cast(i) * 1.5f); + } + auto* largeArray = DataArray::Create(ds, k_LargeArrayName, std::move(largeStore), largeAM->getId()); + REQUIRE(largeArray != nullptr); + + return ds; +} + +/** + * @brief Writes the given DataStructure to a temp .dream3d file and returns the path. + */ +fs::path WriteTestFile(const DataStructure& ds, const std::string& fileName) +{ + fs::path outputPath = GetTestOutputDir() / fileName; + Result<> writeResult = DREAM3D::WriteFile(outputPath, ds); + SIMPLNX_RESULT_REQUIRE_VALID(writeResult); + REQUIRE(fs::exists(outputPath)); + return outputPath; +} + +/** + * @brief Creates a DataStructure for the multi-group prune test: + * GroupA / AttrMatA / ArrayA1 (20 x int32) + * / ArrayA2 (20 x int32) + * GroupB / AttrMatB / ArrayB1 (20 x float32) + * / ArrayB2 (20 x float32) + */ +DataStructure CreateMultiGroupTestDataStructure() +{ + DataStructure ds; + + auto* groupA = DataGroup::Create(ds, k_GroupAName); + REQUIRE(groupA != nullptr); + auto* attrMatA = AttributeMatrix::Create(ds, k_AttrMatAName, {k_PruneArraySize}, groupA->getId()); + REQUIRE(attrMatA != nullptr); + + auto storeA1 = std::make_unique>(std::vector{k_PruneArraySize}, std::vector{1}, static_cast(0)); + for(usize i = 0; i < k_PruneArraySize; ++i) + { + storeA1->setValue(i, static_cast(i)); + } + auto* arrayA1 = DataArray::Create(ds, k_ArrayA1Name, std::move(storeA1), attrMatA->getId()); + REQUIRE(arrayA1 != nullptr); + + auto storeA2 = std::make_unique>(std::vector{k_PruneArraySize}, std::vector{1}, static_cast(0)); + for(usize i = 0; i < k_PruneArraySize; ++i) + { + storeA2->setValue(i, static_cast(i * 2)); + } + auto* arrayA2 = DataArray::Create(ds, k_ArrayA2Name, std::move(storeA2), attrMatA->getId()); + REQUIRE(arrayA2 != nullptr); + + auto* groupB = DataGroup::Create(ds, k_GroupBName); + REQUIRE(groupB != nullptr); + auto* attrMatB = AttributeMatrix::Create(ds, k_AttrMatBName, {k_PruneArraySize}, groupB->getId()); + REQUIRE(attrMatB != nullptr); + + auto storeB1 = std::make_unique>(std::vector{k_PruneArraySize}, std::vector{1}, static_cast(0)); + for(usize i = 0; i < k_PruneArraySize; ++i) + { + storeB1->setValue(i, static_cast(i) * 0.5f); + } + auto* arrayB1 = DataArray::Create(ds, k_ArrayB1Name, std::move(storeB1), attrMatB->getId()); + REQUIRE(arrayB1 != nullptr); + + auto storeB2 = std::make_unique>(std::vector{k_PruneArraySize}, std::vector{1}, static_cast(0)); + for(usize i = 0; i < k_PruneArraySize; ++i) + { + storeB2->setValue(i, static_cast(i) * 0.25f); + } + auto* arrayB2 = DataArray::Create(ds, k_ArrayB2Name, std::move(storeB2), attrMatB->getId()); + REQUIRE(arrayB2 != nullptr); + + return ds; +} +} // namespace + +// ============================================================================= +// Test Scenarios +// ============================================================================= + +TEST_CASE("Dream3dLoadingApi: LoadDataStructure loads all arrays") +{ + DataStructure srcDs = CreateSimpleTestDataStructure(); + fs::path filePath = WriteTestFile(srcDs, "Dream3dLoadingApiTest_LoadAll.dream3d"); + + Result result = DREAM3D::LoadDataStructure(filePath); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + const DataStructure& ds = result.value(); + + const auto& smallPath = k_SmallArrayPath; + const auto& largePath = k_LargeArrayPath; + + // Both arrays must exist + auto* smallArray = ds.getDataAs(smallPath); + REQUIRE(smallArray != nullptr); + REQUIRE(smallArray->getNumberOfTuples() == k_SmallArraySize); + + auto* largeArray = ds.getDataAs(largePath); + REQUIRE(largeArray != nullptr); + REQUIRE(largeArray->getNumberOfTuples() == k_LargeArraySize); + + // Verify SmallArray values + const auto& smallStore = smallArray->getDataStoreRef(); + for(usize i = 0; i < k_SmallArraySize; ++i) + { + CHECK(smallStore[i] == static_cast(i * 3)); + } + + // Verify LargeArray values + const auto& largeStore = largeArray->getDataStoreRef(); + for(usize i = 0; i < k_LargeArraySize; ++i) + { + CHECK(largeStore[i] == Approx(static_cast(i) * 1.5f)); + } +} + +TEST_CASE("Dream3dLoadingApi: LoadDataStructureArrays loads only requested arrays") +{ + DataStructure srcDs = CreateSimpleTestDataStructure(); + fs::path filePath = WriteTestFile(srcDs, "Dream3dLoadingApiTest_Selective.dream3d"); + + const auto& smallPath = k_SmallArrayPath; + const auto& largePath = k_LargeArrayPath; + + Result result = DREAM3D::LoadDataStructureArrays(filePath, {smallPath}); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + const DataStructure& ds = result.value(); + + // SmallArray must exist with correct data + auto* smallArray = ds.getDataAs(smallPath); + REQUIRE(smallArray != nullptr); + REQUIRE(smallArray->getNumberOfTuples() == k_SmallArraySize); + + const auto& smallStore = smallArray->getDataStoreRef(); + for(usize i = 0; i < k_SmallArraySize; ++i) + { + CHECK(smallStore[i] == static_cast(i * 3)); + } + + // LargeArray must NOT exist + auto* largeArray = ds.getDataAs(largePath); + CHECK(largeArray == nullptr); + + // Ancestor containers must exist + CHECK(ds.getDataAs(k_GroupPath) != nullptr); + CHECK(ds.getDataAs(k_SmallAMPath) != nullptr); +} + +TEST_CASE("Dream3dLoadingApi: LoadDataStructureArraysMetadata loads only requested metadata") +{ + DataStructure srcDs = CreateSimpleTestDataStructure(); + fs::path filePath = WriteTestFile(srcDs, "Dream3dLoadingApiTest_SelectiveMeta.dream3d"); + + const auto& smallPath = k_SmallArrayPath; + const auto& largePath = k_LargeArrayPath; + + Result result = DREAM3D::LoadDataStructureArraysMetadata(filePath, {smallPath}); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + const DataStructure& ds = result.value(); + + // SmallArray must exist + auto* smallArray = ds.getDataAs(smallPath); + REQUIRE(smallArray != nullptr); + + // SmallArray store must be Empty (no data loaded) + CHECK(smallArray->getStoreType() == IDataStore::StoreType::Empty); + + // LargeArray must NOT exist + auto* largeArray = ds.getDataAs(largePath); + CHECK(largeArray == nullptr); + + // Ancestor containers must exist + CHECK(ds.getDataAs(k_GroupPath) != nullptr); + CHECK(ds.getDataAs(k_SmallAMPath) != nullptr); +} + +TEST_CASE("Dream3dLoadingApi: LoadDataStructureMetadata loads all metadata") +{ + DataStructure srcDs = CreateSimpleTestDataStructure(); + fs::path filePath = WriteTestFile(srcDs, "Dream3dLoadingApiTest_AllMeta.dream3d"); + + Result result = DREAM3D::LoadDataStructureMetadata(filePath); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + const DataStructure& ds = result.value(); + + const auto& smallPath = k_SmallArrayPath; + const auto& largePath = k_LargeArrayPath; + + // Both arrays must exist + auto* smallArray = ds.getDataAs(smallPath); + REQUIRE(smallArray != nullptr); + auto* largeArray = ds.getDataAs(largePath); + REQUIRE(largeArray != nullptr); + + // Both must have Empty stores (no data loaded) + CHECK(smallArray->getStoreType() == IDataStore::StoreType::Empty); + CHECK(largeArray->getStoreType() == IDataStore::StoreType::Empty); +} + +TEST_CASE("Dream3dLoadingApi: LoadDataStructure with invalid path returns error") +{ + const fs::path bogusPath("/tmp/nonexistent_dream3d_file_12345.dream3d"); + + // Suppress HDF5 error output temporarily + H5Eset_auto(H5E_DEFAULT, nullptr, nullptr); + Result result = DREAM3D::LoadDataStructure(bogusPath); + // Restore default error handling + H5Eset_auto(H5E_DEFAULT, (H5E_auto_t)H5Eprint, stderr); + + REQUIRE(result.invalid()); + REQUIRE(!result.errors().empty()); + CHECK(result.errors()[0].code == -1); +} + +TEST_CASE("Dream3dLoadingApi: LoadDataStructure with legacy file") +{ + const fs::path legacyPath = fs::path(unit_test::k_SourceDir.view()) / "test" / "Data" / "LegacyData.dream3d"; + REQUIRE(fs::exists(legacyPath)); + + Result result = DREAM3D::LoadDataStructure(legacyPath); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + // Verify the DataStructure has content (legacy files should load successfully) + const DataStructure& ds = result.value(); + CHECK(!ds.getAllDataPaths().empty()); +} + +TEST_CASE("Dream3dLoadingApi: LoadDataStructureArrays prune verification") +{ + DataStructure srcDs = CreateMultiGroupTestDataStructure(); + fs::path filePath = WriteTestFile(srcDs, "Dream3dLoadingApiTest_Prune.dream3d"); + + Result result = DREAM3D::LoadDataStructureArrays(filePath, {k_ArrayA1Path}); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + const DataStructure& ds = result.value(); + + // ArrayA1 and its ancestors must exist + auto* arrayA1 = ds.getDataAs(k_ArrayA1Path); + REQUIRE(arrayA1 != nullptr); + REQUIRE(arrayA1->getNumberOfTuples() == k_PruneArraySize); + CHECK(ds.getDataAs(k_GroupAPath) != nullptr); + CHECK(ds.getDataAs(k_AttrMatAPath) != nullptr); + + // ArrayA2 must NOT exist (same group, but not requested) + CHECK(ds.getDataAs(k_ArrayA2Path) == nullptr); + + // GroupB and its children must NOT exist + CHECK(ds.getDataAs(k_GroupBPath) == nullptr); + CHECK(ds.getDataAs(k_ArrayB1Path) == nullptr); + CHECK(ds.getDataAs(k_ArrayB2Path) == nullptr); +} + +TEST_CASE("Dream3dLoadingApi: Recovery file with all in-core data") +{ + DataStructure srcDs = CreateSimpleTestDataStructure(); + fs::path filePath = GetTestOutputDir() / "Dream3dLoadingApiTest_Recovery.dream3d"; + + // Write as a recovery file — without OOC plugin, this behaves like WriteFile + // but exercises the recovery write path + Result<> writeResult = DREAM3D::WriteRecoveryFile(filePath, srcDs); + SIMPLNX_RESULT_REQUIRE_VALID(writeResult); + REQUIRE(fs::exists(filePath)); + + Result result = DREAM3D::LoadDataStructure(filePath); + SIMPLNX_RESULT_REQUIRE_VALID(result); + + const DataStructure& ds = result.value(); + + const auto& smallPath = k_SmallArrayPath; + const auto& largePath = k_LargeArrayPath; + + // All arrays should be in-core since no OOC plugin is loaded + auto* smallArray = ds.getDataAs(smallPath); + REQUIRE(smallArray != nullptr); + CHECK(smallArray->getStoreType() == IDataStore::StoreType::InMemory); + + auto* largeArray = ds.getDataAs(largePath); + REQUIRE(largeArray != nullptr); + CHECK(largeArray->getStoreType() == IDataStore::StoreType::InMemory); + + // Verify data integrity through the recovery round-trip + const auto& smallStore = ds.getDataRefAs(smallPath).getDataStoreRef(); + for(usize i = 0; i < k_SmallArraySize; ++i) + { + CHECK(smallStore[i] == static_cast(i * 3)); + } +} diff --git a/test/H5Test.cpp b/test/H5Test.cpp index aa3d9bdcb1..1235242a61 100644 --- a/test/H5Test.cpp +++ b/test/H5Test.cpp @@ -628,7 +628,7 @@ TEST_CASE("Read Legacy DREAM3D-NX Data") std::filesystem::path filepath = GetLegacyFilepath(); REQUIRE(exists(filepath)); { - Result result = DREAM3D::ImportDataStructureFromFile(filepath, true); + Result result = DREAM3D::LoadDataStructureMetadata(filepath); SIMPLNX_RESULT_REQUIRE_VALID(result); DataStructure dataStructure = result.value(); @@ -1038,9 +1038,7 @@ TEST_CASE("DataStructureAppend") Result<> writeResult = DREAM3D::WriteFile(outputFilePath, baseDataStructure); SIMPLNX_RESULT_REQUIRE_VALID(writeResult); - auto readResult = DREAM3D::ImportDataStructureFromFile(inputFilePath, false); - SIMPLNX_RESULT_REQUIRE_VALID(readResult); - DataStructure exemplarDataStructure = std::move(readResult.value()); + DataStructure exemplarDataStructure = UnitTest::LoadDataStructure(inputFilePath); usize currentTopLevelSize = baseDataStructure.getTopLevelData().size(); for(const DataObject* object : exemplarDataStructure.getTopLevelData()) @@ -1050,10 +1048,7 @@ TEST_CASE("DataStructureAppend") auto appendResult = DREAM3D::AppendFile(outputFilePath, exemplarDataStructure, path); SIMPLNX_RESULT_REQUIRE_VALID(appendResult); - auto appendedFileReadResult = DREAM3D::ImportDataStructureFromFile(outputFilePath, false); - SIMPLNX_RESULT_REQUIRE_VALID(appendedFileReadResult); - - DataStructure appendedDataStructure = std::move(appendedFileReadResult.value()); + DataStructure appendedDataStructure = UnitTest::LoadDataStructure(outputFilePath); currentTopLevelSize++; diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp index ca7cfac790..eb7d7ec45b 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp @@ -1,7 +1,5 @@ #include "UnitTestCommon.hpp" -#include "simplnx/Parameters/Dream3dImportParameter.hpp" - #include #include @@ -12,37 +10,19 @@ namespace nx::core::UnitTest { DataStructure LoadDataStructure(const fs::path& filepath) { - // Ensure the plugins a loaded. LoadPlugins(); - - INFO(fmt::format("Error loading file: '{}' ", filepath.string())); REQUIRE(fs::exists(filepath)); - DataStructure dataStructure; - - // const Uuid k_SimplnxCorePluginId = *Uuid::FromString("05cc618b-781f-4ac0-b9ac-43f26ce1854f"); - auto* filterList = Application::Instance()->getFilterList(); - /************************************************************************* - * ReadDREAM3DFilter - ************************************************************************/ - constexpr Uuid k_ReadDREAM3DFilterId = *Uuid::FromString("0dbd31c7-19e0-4077-83ef-f4a6459a0e2d"); - const FilterHandle k_ReadDREAM3DFilterHandle(k_ReadDREAM3DFilterId, k_SimplnxCorePluginId); - - auto filterPtr = filterList->createFilter(k_ReadDREAM3DFilterHandle); - REQUIRE(nullptr != filterPtr); - - Arguments args; - args.insertOrAssign("import_data_object", std::make_any(Dream3dImportParameter::ImportData{filepath, Dream3dImportParameter::PathImportPolicy::All})); - - // Preflight the filter and check result - auto preflightResult = filterPtr->preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) - - // Execute the filter and check the result - auto executeResult = filterPtr->execute(dataStructure, args); //, nullptr, IFilter::MessageHandler{[](const IFilter::Message& message) { fmt::print("{}\n", message.message); }}); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - - return dataStructure; + auto result = DREAM3D::LoadDataStructure(filepath); + if(result.invalid()) + { + for(const auto& error : result.errors()) + { + UNSCOPED_INFO(fmt::format("[{}] {}", error.code, error.message)); + } + FAIL(fmt::format("Failed to load DataStructure from '{}'", filepath.string())); + } + return std::move(result.value()); } TestFileSentinel::TestFileSentinel(std::string testFilesDir, std::string inputArchiveName, std::string expectedTopLevelOutput, bool decompressFiles, bool removeTemp) From 6adc885cf754c7e3d0e26a28813ecc022fe938ab Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Thu, 9 Apr 2026 22:46:24 -0400 Subject: [PATCH 06/13] STYLE: Use namespace fs = std::filesystem alias across changed files Add the namespace fs = std::filesystem alias to .cpp files that spell out std::filesystem, consistent with the existing convention used throughout the codebase (e.g., AtomicFile.cpp, FileUtilities.cpp, all ITK test files, UnitTestCommon.hpp). Files updated: Dream3dIO.cpp, ImportH5ObjectPathsAction.cpp, DataIOCollection.cpp, H5Test.cpp, UnitTestCommon.cpp, DREAM3DFileTest.cpp, ComputeIPFColorsTest.cpp. --- .../test/ComputeIPFColorsTest.cpp | 2 +- .../SimplnxCore/test/DREAM3DFileTest.cpp | 2 +- .../IO/Generic/DataIOCollection.cpp | 6 ++-- .../Actions/ImportH5ObjectPathsAction.cpp | 3 +- .../Utilities/Parsing/DREAM3D/Dream3dIO.cpp | 31 ++++++++++--------- test/H5Test.cpp | 10 +++--- .../simplnx/UnitTest/UnitTestCommon.cpp | 8 ++--- 7 files changed, 33 insertions(+), 29 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp b/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp index 3c21e43f1a..8e4e937797 100644 --- a/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp @@ -111,7 +111,7 @@ TEST_CASE("OrientationAnalysis::ComputeIPFColors", "[OrientationAnalysis][Comput SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); { // Write out the DataStructure for later viewing/debugging - auto fileWriter = nx::core::HDF5::FileIO::WriteFile(std::filesystem::path(fmt::format("{}/ComputeIPFColors_Test.dream3d", unit_test::k_BinaryTestOutputDir))); + auto fileWriter = nx::core::HDF5::FileIO::WriteFile(fs::path(fmt::format("{}/ComputeIPFColors_Test.dream3d", unit_test::k_BinaryTestOutputDir))); auto resultH5 = HDF5::DataStructureWriter::WriteFile(dataStructure, fileWriter); SIMPLNX_RESULT_REQUIRE_VALID(resultH5); } diff --git a/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp b/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp index 367fa4aedb..0bafd8e8b9 100644 --- a/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp +++ b/src/Plugins/SimplnxCore/test/DREAM3DFileTest.cpp @@ -65,7 +65,7 @@ const FilterHandle k_ImportD3DHandle(Uuid::FromString("0dbd31c7-19e0-4077-83ef-f fs::path GetDataDir(const Application& app) { - return std::filesystem::path(unit_test::k_BinaryTestOutputDir.view()); + return fs::path(unit_test::k_BinaryTestOutputDir.view()); } fs::path GetIODataPath() diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index 2513318b7b..31b0190844 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -9,6 +9,8 @@ #include "simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" +namespace fs = std::filesystem; + namespace nx::core { DataIOCollection::DataIOCollection() @@ -101,7 +103,7 @@ bool DataIOCollection::hasReadOnlyRefCreationFnc(const std::string& type) const return false; } -std::unique_ptr DataIOCollection::createReadOnlyRefStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, +std::unique_ptr DataIOCollection::createReadOnlyRefStore(const std::string& type, DataType numericType, const fs::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& componentShape, const ShapeType& chunkShape) { // Find the first IO manager that has a factory for the requested format and @@ -138,7 +140,7 @@ bool DataIOCollection::hasReadOnlyRefListCreationFnc(const std::string& type) co return false; } -std::unique_ptr DataIOCollection::createReadOnlyRefListStore(const std::string& type, DataType numericType, const std::filesystem::path& filePath, const std::string& datasetPath, +std::unique_ptr DataIOCollection::createReadOnlyRefListStore(const std::string& type, DataType numericType, const fs::path& filePath, const std::string& datasetPath, const ShapeType& tupleShape, const ShapeType& chunkShape) { // Find the first IO manager that has a ListStore reference factory for the diff --git a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp index 28e28aeb0b..87ff1dce5f 100644 --- a/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp +++ b/src/simplnx/Filter/Actions/ImportH5ObjectPathsAction.cpp @@ -10,10 +10,11 @@ #include using namespace nx::core; +namespace fs = std::filesystem; namespace nx::core { -ImportH5ObjectPathsAction::ImportH5ObjectPathsAction(const std::filesystem::path& importFile, const PathsType& paths) +ImportH5ObjectPathsAction::ImportH5ObjectPathsAction(const fs::path& importFile, const PathsType& paths) : IDataCreationAction(DataPath{}) , m_H5FilePath(importFile) , m_Paths(paths) diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp index 05d445ab31..af84d03e72 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp @@ -38,6 +38,7 @@ #include using namespace nx::core; +namespace fs = std::filesystem; namespace { @@ -749,14 +750,14 @@ void WriteXdmf(std::ostream& out, const DataStructure& dataStructure, std::strin } } // namespace -void DREAM3D::WriteXdmf(const std::filesystem::path& filePath, const DataStructure& dataStructure, std::string_view hdf5FilePath) +void DREAM3D::WriteXdmf(const fs::path& filePath, const DataStructure& dataStructure, std::string_view hdf5FilePath) { std::ofstream file(filePath); ::WriteXdmf(file, dataStructure, hdf5FilePath); } -DREAM3D::FileVersionType DREAM3D::GetFileVersion(const std::filesystem::path& path) +DREAM3D::FileVersionType DREAM3D::GetFileVersion(const fs::path& path) { auto fileReader = HDF5::FileIO::ReadFile(path); return GetFileVersion(fileReader); @@ -2257,9 +2258,9 @@ Result DREAM3D::ImportPipelineJsonFromFile(const nx::core::HDF5: return {nlohmann::json::parse(pipelineJsonString)}; } -Result DREAM3D::ImportPipelineFromFile(const std::filesystem::path& filePath) +Result DREAM3D::ImportPipelineFromFile(const fs::path& filePath) { - if(!std::filesystem::exists(filePath)) + if(!fs::exists(filePath)) { return MakeErrorResult(-1, fmt::format("DREAM3D::ImportPipelineFromFile: File does not exist. '{}'", filePath.string())); } @@ -2272,9 +2273,9 @@ Result DREAM3D::ImportPipelineFromFile(const std::filesystem::path& fi return ImportPipelineFromFile(fileReader); } -Result DREAM3D::ImportPipelineJsonFromFile(const std::filesystem::path& filePath) +Result DREAM3D::ImportPipelineJsonFromFile(const fs::path& filePath) { - if(!std::filesystem::exists(filePath)) + if(!fs::exists(filePath)) { return MakeErrorResult(-1, fmt::format("DREAM3D::ImportPipelineFromFile: File does not exist. '{}'", filePath.string())); } @@ -2340,7 +2341,7 @@ Result<> DREAM3D::WriteFile(nx::core::HDF5::FileIO& fileWriter, const Pipeline& return WriteDataStructure(fileWriter, dataStructure); } -Result<> DREAM3D::WriteFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline, bool writeXdmf) +Result<> DREAM3D::WriteFile(const fs::path& path, const DataStructure& dataStructure, const Pipeline& pipeline, bool writeXdmf) { auto fileWriter = nx::core::HDF5::FileIO::WriteFile(path); if(!fileWriter.isValid()) @@ -2356,14 +2357,14 @@ Result<> DREAM3D::WriteFile(const std::filesystem::path& path, const DataStructu if(writeXdmf) { - std::filesystem::path xdmfFilePath = std::filesystem::path(path).replace_extension(".xdmf"); + fs::path xdmfFilePath = fs::path(path).replace_extension(".xdmf"); WriteXdmf(xdmfFilePath, dataStructure, path.filename().string()); } return {}; } -Result<> DREAM3D::WriteRecoveryFile(const std::filesystem::path& path, const DataStructure& dataStructure, const Pipeline& pipeline) +Result<> DREAM3D::WriteRecoveryFile(const fs::path& path, const DataStructure& dataStructure, const Pipeline& pipeline) { // Obtain the global DataIOCollection so we can activate the write-array-override. // The SimplnxOoc plugin registers a callback on this collection at startup that @@ -2382,7 +2383,7 @@ Result<> DREAM3D::WriteRecoveryFile(const std::filesystem::path& path, const Dat return WriteFile(path, dataStructure, pipeline, false); } -Result<> DREAM3D::AppendFile(const std::filesystem::path& path, const DataStructure& dataStructure, const DataPath& dataPath) +Result<> DREAM3D::AppendFile(const fs::path& path, const DataStructure& dataStructure, const DataPath& dataPath) { auto file = nx::core::HDF5::FileIO::AppendFile(path); if(!file.isValid()) @@ -2605,7 +2606,7 @@ void PruneDataStructure(DataStructure& ds, const std::vector& keepPath * 3. Insert each object via LoadDataObjectFromHDF5 (shallow copy + insert + optional data load) * 4. If a handler is registered, run it for deferred loading */ -Result LoadDataStructureWithHandler(const std::filesystem::path& filePath, const std::vector& paths) +Result LoadDataStructureWithHandler(const fs::path& filePath, const std::vector& paths) { auto fileReader = nx::core::HDF5::FileIO::ReadFile(filePath); if(!fileReader.isValid()) @@ -2668,7 +2669,7 @@ Result LoadDataStructureWithHandler(const std::filesystem::path& // New public LoadDataStructure* API // --------------------------------------------------------------------------- -Result DREAM3D::LoadDataStructureMetadata(const std::filesystem::path& path) +Result DREAM3D::LoadDataStructureMetadata(const fs::path& path) { auto fileReader = nx::core::HDF5::FileIO::ReadFile(path); if(!fileReader.isValid()) @@ -2678,7 +2679,7 @@ Result DREAM3D::LoadDataStructureMetadata(const std::filesystem:: return LoadDataStructureMetadataInternal(fileReader); } -Result DREAM3D::LoadDataStructure(const std::filesystem::path& path) +Result DREAM3D::LoadDataStructure(const fs::path& path) { auto metadataResult = DREAM3D::LoadDataStructureMetadata(path); if(metadataResult.invalid()) @@ -2689,7 +2690,7 @@ Result DREAM3D::LoadDataStructure(const std::filesystem::path& pa return LoadDataStructureWithHandler(path, allPaths); } -Result DREAM3D::LoadDataStructureArrays(const std::filesystem::path& path, const std::vector& dataPaths) +Result DREAM3D::LoadDataStructureArrays(const fs::path& path, const std::vector& dataPaths) { auto result = LoadDataStructureWithHandler(path, dataPaths); if(result.invalid()) @@ -2700,7 +2701,7 @@ Result DREAM3D::LoadDataStructureArrays(const std::filesystem::pa return result; } -Result DREAM3D::LoadDataStructureArraysMetadata(const std::filesystem::path& path, const std::vector& dataPaths) +Result DREAM3D::LoadDataStructureArraysMetadata(const fs::path& path, const std::vector& dataPaths) { auto result = DREAM3D::LoadDataStructureMetadata(path); if(result.invalid()) diff --git a/test/H5Test.cpp b/test/H5Test.cpp index 1235242a61..51ed5a2f59 100644 --- a/test/H5Test.cpp +++ b/test/H5Test.cpp @@ -57,13 +57,13 @@ const fs::path k_ComplexH5File = "new.h5"; fs::path GetDataDir() { - return std::filesystem::path(unit_test::k_BinaryTestOutputDir.view()); + return fs::path(unit_test::k_BinaryTestOutputDir.view()); } fs::path GetLegacyFilepath() { std::string path = fmt::format("{}/test/Data/{}", unit_test::k_SourceDir.view(), Constants::k_LegacyFilepath); - return std::filesystem::path(path); + return fs::path(path); } fs::path GetComplexH5File() @@ -625,7 +625,7 @@ H5ClassT TestH5ImplicitCopy(H5ClassT&& originalObject, std::string_view testedCl TEST_CASE("Read Legacy DREAM3D-NX Data") { auto app = Application::GetOrCreateInstance(); - std::filesystem::path filepath = GetLegacyFilepath(); + fs::path filepath = GetLegacyFilepath(); REQUIRE(exists(filepath)); { Result result = DREAM3D::LoadDataStructureMetadata(filepath); @@ -1024,8 +1024,8 @@ TEST_CASE("HDF5ImplicitCopyIOTest") TEST_CASE("DataStructureAppend") { - const std::filesystem::path inputFilePath = fs::path(unit_test::k_SourceDir.view()) / "test/Data/geoms.dream3d"; - const std::filesystem::path outputFilePath = GetDataDir() / "DataStructureAppend.dream3d"; + const fs::path inputFilePath = fs::path(unit_test::k_SourceDir.view()) / "test/Data/geoms.dream3d"; + const fs::path outputFilePath = GetDataDir() / "DataStructureAppend.dream3d"; const DataPath originalArrayPath({"foo"}); DataStructure baseDataStructure; diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp index eb7d7ec45b..bfdc9709d9 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.cpp @@ -49,7 +49,7 @@ TestFileSentinel::~TestFileSentinel() if(m_RemoveTemp) { std::error_code errorCode; - std::filesystem::remove_all(fmt::format("{}/{}", m_TestFilesDir, m_ExpectedTopLevelOutput), errorCode); + fs::remove_all(fmt::format("{}/{}", m_TestFilesDir, m_ExpectedTopLevelOutput), errorCode); if(errorCode) { std::cout << "Removing decompressed data failed: " << errorCode.message() << std::endl; @@ -222,13 +222,13 @@ std::error_code TestFileSentinel::decompress() // typeFlag: '5' = directory, '0' or '\0' = regular file, '2' = symlink if(typeFlag == '5') { - std::filesystem::create_directories(fullPath); + fs::create_directories(fullPath); } else if(typeFlag == '0' || typeFlag == '\0') { // Ensure parent directory exists - std::filesystem::path filePath(fullPath); - std::filesystem::create_directories(filePath.parent_path()); + fs::path filePath(fullPath); + fs::create_directories(filePath.parent_path()); std::ofstream outFile(fullPath, std::ios::binary); if(!outFile) From 8a04d6c82ae50ef50595b14c4f7f17731081d0bd Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Fri, 10 Apr 2026 11:12:53 -0400 Subject: [PATCH 07/13] REFACTOR: Make IDataStore::getRecoveryMetadata pure virtual Previously IDataStore provided a default implementation that returned an empty map, which silently disabled recovery metadata for any store subclass that forgot to override it. Make it pure virtual so every concrete store must explicitly state what (if any) recovery metadata it produces. DataStore overrides it to return an empty map (in-memory stores have no backing file or external state, so the recovery file's HDF5 dataset contains all the data needed to reconstruct the store). EmptyDataStore overrides it to throw std::runtime_error, matching the fail-fast behavior of every other data-access method on this metadata- only placeholder class. Querying recovery metadata on a placeholder is a programming error: the real store that replaces the placeholder during execution is the one responsible for providing recovery info. MockOocDataStore in IParallelAlgorithmTest.cpp gains a no-op override returning an empty map so it remains constructible. Signed-off-by: Joey Kleingers --- src/simplnx/DataStructure/DataStore.hpp | 21 ++++++++++++++++++++ src/simplnx/DataStructure/EmptyDataStore.hpp | 20 +++++++++++++++++++ src/simplnx/DataStructure/IDataStore.hpp | 13 +++++------- test/IParallelAlgorithmTest.cpp | 7 +++++++ 4 files changed, 53 insertions(+), 8 deletions(-) diff --git a/src/simplnx/DataStructure/DataStore.hpp b/src/simplnx/DataStructure/DataStore.hpp index bcccd1847f..0b3e44ebfa 100644 --- a/src/simplnx/DataStructure/DataStore.hpp +++ b/src/simplnx/DataStructure/DataStore.hpp @@ -11,10 +11,12 @@ #include #include #include +#include #include #include #include #include +#include #include namespace nx::core @@ -208,6 +210,25 @@ class DataStore : public AbstractDataStore return IDataStore::StoreType::InMemory; } + /** + * @brief Returns recovery metadata for an in-memory store. + * + * In-memory DataStores have no backing file or external state, so the + * recovery file's HDF5 dataset for this array contains all the data + * needed to reconstruct the store. No extra key-value attributes are + * required, so this returns an empty map. + * + * Out-of-core store subclasses override this to return the file path, + * dataset path, chunk shape, etc. needed to reattach to their backing + * storage after a crash. + * + * @return std::map Empty map. + */ + std::map getRecoveryMetadata() const override + { + return {}; + } + /** * @brief This method copies a value to the member variable m_InitValue */ diff --git a/src/simplnx/DataStructure/EmptyDataStore.hpp b/src/simplnx/DataStructure/EmptyDataStore.hpp index 259e3b303f..14f8b4c315 100644 --- a/src/simplnx/DataStructure/EmptyDataStore.hpp +++ b/src/simplnx/DataStructure/EmptyDataStore.hpp @@ -4,8 +4,10 @@ #include +#include #include #include +#include #include namespace nx::core @@ -120,6 +122,24 @@ class EmptyDataStore : public AbstractDataStore return IDataStore::StoreType::Empty; } + /** + * @brief Throws — EmptyDataStore is a metadata-only placeholder. + * + * EmptyDataStore holds no data and no backing file, so it has no + * recovery metadata to report. Calling getRecoveryMetadata() on an + * EmptyDataStore is a programming error: the caller is treating a + * placeholder as if it were a real store. The real store that + * replaces this placeholder during execution is the one responsible + * for providing recovery metadata. + * + * Throws std::runtime_error to fail fast, matching the behavior of + * the other data-access methods on this class. + */ + std::map getRecoveryMetadata() const override + { + throw std::runtime_error("EmptyDataStore::getRecoveryMetadata: cannot query recovery metadata on a placeholder store"); + } + /** * @brief Returns the data format string that was specified at construction. * diff --git a/src/simplnx/DataStructure/IDataStore.hpp b/src/simplnx/DataStructure/IDataStore.hpp index c32e54db3c..dcc250cad7 100644 --- a/src/simplnx/DataStructure/IDataStore.hpp +++ b/src/simplnx/DataStructure/IDataStore.hpp @@ -145,10 +145,10 @@ class SIMPLNX_EXPORT IDataStore * data is written directly into the recovery file's HDF5 datasets; * no extra metadata is required. * - * **Out-of-core stores** override this to return key-value pairs - * describing their backing file path, HDF5 dataset path, chunk - * shape, and any other parameters needed to reconstruct the OOC - * store from the file on disk. + * **Out-of-core stores** return key-value pairs describing their + * backing file path, HDF5 dataset path, chunk shape, and any other + * parameters needed to reconstruct the OOC store from the file on + * disk. * * Each key-value pair is written as an HDF5 string attribute on the * array's dataset inside the recovery file. The recovery loader @@ -158,10 +158,7 @@ class SIMPLNX_EXPORT IDataStore * @return std::map Key-value pairs of * recovery metadata. Empty for in-memory stores. */ - virtual std::map getRecoveryMetadata() const - { - return {}; - } + virtual std::map getRecoveryMetadata() const = 0; /** * @brief Returns the size of the stored type of the data store. diff --git a/test/IParallelAlgorithmTest.cpp b/test/IParallelAlgorithmTest.cpp index 426b213ec5..8d99291049 100644 --- a/test/IParallelAlgorithmTest.cpp +++ b/test/IParallelAlgorithmTest.cpp @@ -5,8 +5,10 @@ #include +#include #include #include +#include using namespace nx::core; @@ -39,6 +41,11 @@ class MockOocDataStore : public AbstractDataStore return IDataStore::StoreType::OutOfCore; } + std::map getRecoveryMetadata() const override + { + return {}; + } + usize getNumberOfTuples() const override { return m_NumTuples; From 9fe576e5cdc7b52eb91affcb5e807f20eea97d58 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Fri, 10 Apr 2026 22:34:04 -0400 Subject: [PATCH 08/13] REFACTOR: Return reference from GetIOCollection and clean up in-memory format sentinel Addresses code review feedback on DataIOCollection ownership and factory error messages. Ownership clarification: * DataStoreUtilities::GetIOCollection() and Application::getIOCollection() now return DataIOCollection& instead of std::shared_ptr. The collection is owned by the Application singleton which outlives every caller, so a reference expresses non-ownership more clearly than a shared_ptr and prevents accidental lifetime extension. * WriteArrayOverrideGuard stores a DataIOCollection& member. Since the guard is already non-copyable and non-movable, a reference member is natural and the "may be null no-op" path was dropped (no caller used it). In-memory format sentinel hygiene: * CoreDataIOManager::formatName() now returns Preferences::k_InMemoryFormat instead of the empty string. Empty means "unset/auto" and k_InMemoryFormat means "explicit in-memory"; previously "" was doing double duty. * DataIOCollection constructor registers the core manager directly into the manager map, bypassing the addIOManager() guard. The guard still rejects plugin registrations under the reserved name. * createDataStore/createListStore fallbacks now look up the core manager from m_ManagerMap under k_InMemoryFormat instead of constructing a fresh local CoreDataIOManager. * ArrayCreationUtilities no longer translates k_InMemoryFormat to ""; the RAM-check path recognizes both sentinels as in-core. Actionable factory errors: * Added DataIOCollection::generateManagerListString() that produces a padded multi-line capability matrix of every registered IO manager and the store types it supports (DataStore, ListStore, StringStore, ReadOnlyRef(DataStore), ReadOnlyRef(ListStore)). Uses display names where registered, falling back to the raw format identifier. * Wired the helper into the existing CreateArray nullptr-check error message so users can immediately see which formats are available when a requested format is unknown. Tests updated to reflect the new reference API. Signed-off-by: Joey Kleingers --- src/simplnx/Core/Application.cpp | 4 +- src/simplnx/Core/Application.hpp | 9 +- .../IO/Generic/CoreDataIOManager.cpp | 7 +- .../IO/Generic/DataIOCollection.cpp | 114 +++++++++++++++++- .../IO/Generic/DataIOCollection.hpp | 48 +++++--- .../IO/HDF5/DataStructureWriter.cpp | 6 +- .../Utilities/ArrayCreationUtilities.hpp | 34 +++--- src/simplnx/Utilities/DataStoreUtilities.cpp | 2 +- src/simplnx/Utilities/DataStoreUtilities.hpp | 37 ++++-- .../Utilities/Parsing/DREAM3D/Dream3dIO.cpp | 7 +- test/AppTest.cpp | 6 - test/IOFormat.cpp | 16 +-- 12 files changed, 215 insertions(+), 75 deletions(-) diff --git a/src/simplnx/Core/Application.cpp b/src/simplnx/Core/Application.cpp index 8a28cd855b..2b19b18de0 100644 --- a/src/simplnx/Core/Application.cpp +++ b/src/simplnx/Core/Application.cpp @@ -338,9 +338,9 @@ JsonPipelineBuilder* Application::getPipelineBuilder() const return nullptr; } -std::shared_ptr Application::getIOCollection() const +DataIOCollection& Application::getIOCollection() const { - return m_DataIOCollection; + return *m_DataIOCollection; } std::shared_ptr Application::getIOManager(const std::string& formatName) const diff --git a/src/simplnx/Core/Application.hpp b/src/simplnx/Core/Application.hpp index f96939e6b6..a505f31916 100644 --- a/src/simplnx/Core/Application.hpp +++ b/src/simplnx/Core/Application.hpp @@ -122,9 +122,14 @@ class SIMPLNX_EXPORT Application /** * @brief Returns the collection of data I/O managers. - * @return Shared pointer to the DataIOCollection + * + * The returned reference is non-owning; the DataIOCollection is owned by + * the Application singleton and lives for the entire process lifetime. + * Callers should not attempt to extend its lifetime. + * + * @return Reference to the DataIOCollection owned by the Application. */ - std::shared_ptr getIOCollection() const; + DataIOCollection& getIOCollection() const; /** * @brief Returns the I/O manager for the specified format. diff --git a/src/simplnx/DataStructure/IO/Generic/CoreDataIOManager.cpp b/src/simplnx/DataStructure/IO/Generic/CoreDataIOManager.cpp index 4b92eff22a..f56fbd493d 100644 --- a/src/simplnx/DataStructure/IO/Generic/CoreDataIOManager.cpp +++ b/src/simplnx/DataStructure/IO/Generic/CoreDataIOManager.cpp @@ -1,5 +1,6 @@ #include "CoreDataIOManager.hpp" +#include "simplnx/Core/Preferences.hpp" #include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/ListStore.hpp" @@ -17,7 +18,11 @@ CoreDataIOManager::~CoreDataIOManager() noexcept = default; std::string CoreDataIOManager::formatName() const { - return ""; + // The core in-memory manager uses the reserved k_InMemoryFormat constant so + // that "in-memory" is distinct from "unset" (empty string). Callers that want + // explicit in-memory storage should pass k_InMemoryFormat; callers that pass + // "" are signaling "unset/auto — let the resolver decide". + return std::string(Preferences::k_InMemoryFormat); } void CoreDataIOManager::addCoreFactories() diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index 31b0190844..c9ec8d8c79 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -15,7 +15,15 @@ namespace nx::core { DataIOCollection::DataIOCollection() { - addIOManager(std::make_shared()); + // Register the built-in CoreDataIOManager directly into the map. The core + // manager's formatName() is k_InMemoryFormat, which is a reserved name that + // addIOManager() rejects for plugin registrations — so we bypass that + // validation here by writing to m_ManagerMap directly. This is the only + // place in the codebase permitted to register a manager under the reserved + // in-memory format name. + auto coreManager = std::make_shared(); + m_ManagerMap[coreManager->formatName()] = coreManager; + addIOManager(std::make_shared()); } DataIOCollection::~DataIOCollection() noexcept = default; @@ -28,6 +36,9 @@ void DataIOCollection::addIOManager(std::shared_ptr manager) } const std::string& name = manager->formatName(); + // k_InMemoryFormat is reserved for the built-in CoreDataIOManager, which is + // registered directly by the constructor. Any other attempt to register + // under this name (e.g., from a plugin) is rejected. if(name == Preferences::k_InMemoryFormat) { throw std::runtime_error(fmt::format("Cannot register an I/O manager with the reserved format name '{}'", std::string(Preferences::k_InMemoryFormat))); @@ -67,8 +78,11 @@ std::unique_ptr DataIOCollection::createDataStore(const std::string& } } - nx::core::Generic::CoreDataIOManager coreManager; - return coreManager.dataStoreCreationFnc(coreManager.formatName())(dataType, tupleShape, componentShape, {}); + // Fallback: no registered manager claimed @p type, so default to in-memory + // storage. The built-in CoreDataIOManager is always registered under + // k_InMemoryFormat by our constructor, so .at() is guaranteed to succeed. + const auto& coreManager = m_ManagerMap.at(std::string(Preferences::k_InMemoryFormat)); + return coreManager->dataStoreCreationFnc(coreManager->formatName())(dataType, tupleShape, componentShape, {}); } std::unique_ptr DataIOCollection::createListStore(const std::string& type, DataType dataType, const ShapeType& tupleShape) const @@ -81,8 +95,9 @@ std::unique_ptr DataIOCollection::createListStore(const std::string& } } - nx::core::Generic::CoreDataIOManager coreManager; - return coreManager.listStoreCreationFnc(coreManager.formatName())(dataType, tupleShape); + // Fallback: see createDataStore for rationale. Core manager is always present. + const auto& coreManager = m_ManagerMap.at(std::string(Preferences::k_InMemoryFormat)); + return coreManager->listStoreCreationFnc(coreManager->formatName())(dataType, tupleShape); } // --------------------------------------------------------------------------- @@ -297,6 +312,95 @@ std::vector> DataIOCollection::getFormatDisp return result; } +std::string DataIOCollection::generateManagerListString() const +{ + // Build one row per registered manager. Each row pairs the display name + // with a comma-separated list of store-type capabilities. We collect first, + // then format the output as a padded table so columns align. + struct Row + { + std::string displayName; + std::string capabilityList; + }; + std::vector rows; + rows.reserve(m_ManagerMap.size()); + + usize maxNameWidth = 0; + for(const auto& [managerKey, manager] : m_ManagerMap) + { + // The manager's own formatName() is the key under which it registers its + // factories (convention enforced across all known managers). Query with + // that key to determine what this manager can create. + const std::string fn = manager->formatName(); + + // Resolve a friendly display name. The core in-memory manager has a fixed + // label; plugin-registered managers may have a registered display name; + // otherwise fall back to the raw format identifier. + std::string displayName; + if(fn == Preferences::k_InMemoryFormat) + { + displayName = "In Memory"; + } + else + { + auto it = m_FormatDisplayNames.find(fn); + displayName = (it != m_FormatDisplayNames.end()) ? it->second : fn; + } + + // Collect the capability labels in a stable order so rows read consistently. + std::vector capabilities; + if(manager->hasDataStoreCreationFnc(fn)) + { + capabilities.emplace_back("DataStore"); + } + if(manager->hasListStoreCreationFnc(fn)) + { + capabilities.emplace_back("ListStore"); + } + if(manager->hasStringStoreCreationFnc(fn)) + { + capabilities.emplace_back("StringStore"); + } + if(manager->hasReadOnlyRefCreationFnc(fn)) + { + capabilities.emplace_back("ReadOnlyRef(DataStore)"); + } + if(manager->hasListStoreRefCreationFnc(fn)) + { + capabilities.emplace_back("ReadOnlyRef(ListStore)"); + } + + std::string capList; + for(usize i = 0; i < capabilities.size(); ++i) + { + if(i > 0) + { + capList += ", "; + } + capList += capabilities[i]; + } + if(capList.empty()) + { + capList = "(no factories registered)"; + } + + if(displayName.size() > maxNameWidth) + { + maxNameWidth = displayName.size(); + } + rows.push_back({std::move(displayName), std::move(capList)}); + } + + // Assemble the padded output. The leading newline keeps the table from + // butting up against the caller's error-message prefix. + std::string result = "Registered IO managers and their capabilities:"; + for(const auto& row : rows) + { + result += fmt::format("\n {:<{}} : {}", row.displayName, maxNameWidth, row.capabilityList); + } + return result; +} + DataIOCollection::iterator DataIOCollection::begin() { return m_ManagerMap.begin(); diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index d55326cea6..82c985b9e7 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -494,6 +494,28 @@ class SIMPLNX_EXPORT DataIOCollection */ std::vector> getFormatDisplayNames() const; + /** + * @brief Produces a human-readable, multi-line description of every registered + * IO manager and the store types it can create. + * + * Intended for error messages when a createXxxStore() call returns nullptr so + * the user can immediately see which format names are available and what each + * one supports. Each row lists the manager's display name (falling back to its + * format-name identifier) followed by the set of factories it registers: + * DataStore, ListStore, StringStore, ReadOnlyRef(DataStore), ReadOnlyRef(ListStore). + * + * Example output: + * @code + * Registered IO managers and their capabilities: + * In Memory : DataStore, ListStore + * HDF5 : DataStore, ListStore, StringStore + * HDF5-OOC : DataStore, ListStore, StringStore, ReadOnlyRef(DataStore), ReadOnlyRef(ListStore) + * @endcode + * + * @return A multi-line string ready to drop into a fmt::format error message. + */ + std::string generateManagerListString() const; + private: map_type m_ManagerMap; FormatResolverFnc m_FormatResolver; ///< Plugin-provided callback that selects storage format for new arrays @@ -537,18 +559,18 @@ class SIMPLNX_EXPORT WriteArrayOverrideGuard public: /** * @brief Constructs the guard and activates the write-array-override. + * + * The guard stores a reference to the collection, so the collection must + * outlive the guard. In practice the DataIOCollection is owned by the + * Application singleton, which outlives every caller. + * * @param ioCollection The DataIOCollection whose override should be activated. - * May be nullptr, in which case this guard is a no-op. */ - explicit WriteArrayOverrideGuard(std::shared_ptr ioCollection) - : m_IOCollection(std::move(ioCollection)) + explicit WriteArrayOverrideGuard(DataIOCollection& ioCollection) + : m_IOCollection(ioCollection) { // Activate the override so DataStructureWriter will consult the callback. - // Null-check allows callers to conditionally construct the guard. - if(m_IOCollection) - { - m_IOCollection->setWriteArrayOverrideActive(true); - } + m_IOCollection.setWriteArrayOverrideActive(true); } /** @@ -558,19 +580,17 @@ class SIMPLNX_EXPORT WriteArrayOverrideGuard ~WriteArrayOverrideGuard() { // Deactivate the override so subsequent writes go through the normal path. - if(m_IOCollection) - { - m_IOCollection->setWriteArrayOverrideActive(false); - } + m_IOCollection.setWriteArrayOverrideActive(false); } - // Non-copyable and non-movable to prevent scope-escape bugs. + // Non-copyable and non-movable to prevent scope-escape bugs. Additionally + // required because a reference member cannot be rebound or reassigned. WriteArrayOverrideGuard(const WriteArrayOverrideGuard&) = delete; WriteArrayOverrideGuard& operator=(const WriteArrayOverrideGuard&) = delete; WriteArrayOverrideGuard(WriteArrayOverrideGuard&&) = delete; WriteArrayOverrideGuard& operator=(WriteArrayOverrideGuard&&) = delete; private: - std::shared_ptr m_IOCollection; ///< Held alive for the guard's lifetime to ensure safe deactivation + DataIOCollection& m_IOCollection; ///< Non-owning reference; collection must outlive the guard }; } // namespace nx::core diff --git a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp index 85b8d6391c..0fc41e2bd3 100644 --- a/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp +++ b/src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cpp @@ -131,10 +131,10 @@ Result<> DataStructureWriter::writeDataObject(const DataObject* dataObject, nx:: // // If the callback returns std::nullopt the object is not OOC-backed, so // we fall through to the normal HDF5 write path below. - auto ioCollection = DataStoreUtilities::GetIOCollection(); - if(ioCollection->isWriteArrayOverrideActive()) + auto& ioCollection = DataStoreUtilities::GetIOCollection(); + if(ioCollection.isWriteArrayOverrideActive()) { - auto overrideResult = ioCollection->runWriteArrayOverride(*this, dataObject, parentGroup); + auto overrideResult = ioCollection.runWriteArrayOverride(*this, dataObject, parentGroup); if(overrideResult.has_value()) { return overrideResult.value(); diff --git a/src/simplnx/Utilities/ArrayCreationUtilities.hpp b/src/simplnx/Utilities/ArrayCreationUtilities.hpp index 26ef98e794..8c9e7b1f4d 100644 --- a/src/simplnx/Utilities/ArrayCreationUtilities.hpp +++ b/src/simplnx/Utilities/ArrayCreationUtilities.hpp @@ -90,28 +90,25 @@ Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, if(!dataFormat.empty()) { // User explicitly chose a format via the filter UI — bypass the resolver. - // The k_InMemoryFormat sentinel means "force in-memory"; translate it to - // an empty string so the downstream DataStore factory creates an in-core store. - if(dataFormat == Preferences::k_InMemoryFormat.str()) - { - resolvedFormat = ""; - } - else - { - resolvedFormat = dataFormat; - } + // Both k_InMemoryFormat and any other plugin format name (e.g., "HDF5-OOC") + // pass through unchanged; the DataStore factory in DataIOCollection routes + // k_InMemoryFormat to the built-in core manager directly. + resolvedFormat = dataFormat; } else { // No per-filter override — ask the resolver (which consults user preferences, - // size thresholds, and geometry type). - auto ioCollection = DataStoreUtilities::GetIOCollection(); - resolvedFormat = ioCollection->resolveFormat(dataStructure, path, GetDataType(), requiredMemory); + // size thresholds, and geometry type). The resolver returns either "" for + // "default in-memory" or a plugin format name like "HDF5-OOC". + resolvedFormat = DataStoreUtilities::GetIOCollection().resolveFormat(dataStructure, path, GetDataType(), requiredMemory); } // Only check RAM availability for in-core arrays. OOC arrays go to disk - // and do not consume RAM for their primary storage. - if(resolvedFormat.empty() && !CheckMemoryRequirement(dataStructure, requiredMemory)) + // and do not consume RAM for their primary storage. "In-core" means either + // the empty/unset sentinel (resolver defaulted) or the explicit k_InMemoryFormat + // constant (user forced in-memory). + const bool isInCore = resolvedFormat.empty() || resolvedFormat == Preferences::k_InMemoryFormat.str(); + if(isInCore && !CheckMemoryRequirement(dataStructure, requiredMemory)) { uint64 totalMemory = requiredMemory + dataStructure.memoryUsage(); uint64 availableMemory = Memory::GetTotalMemory(); @@ -126,7 +123,12 @@ Result<> CreateArray(DataStructure& dataStructure, const ShapeType& tupleShape, auto store = DataStoreUtilities::CreateDataStore(tupleShape, compShape, mode, resolvedFormat); if(nullptr == store) { - return MakeErrorResult(-265, fmt::format("CreateArray: Unable to create DataStore at '{}' of DataStore format '{}'", path.toString(), resolvedFormat)); + // No registered IO manager could produce a DataStore for this format. + // Include the full manager capability list so the user can tell whether + // the format is a typo, whether the required plugin is missing, or whether + // the format simply does not support this store type. + return MakeErrorResult(-265, fmt::format("CreateArray: Unable to create DataStore at '{}' of DataStore format '{}'.\n{}", path.toString(), resolvedFormat, + DataStoreUtilities::GetIOCollection().generateManagerListString())); } if(!fillValue.empty()) { diff --git a/src/simplnx/Utilities/DataStoreUtilities.cpp b/src/simplnx/Utilities/DataStoreUtilities.cpp index 1c25beda9a..8a10dfc1b6 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.cpp +++ b/src/simplnx/Utilities/DataStoreUtilities.cpp @@ -5,7 +5,7 @@ using namespace nx::core; //----------------------------------------------------------------------------- -std::shared_ptr DataStoreUtilities::GetIOCollection() +DataIOCollection& DataStoreUtilities::GetIOCollection() { return Application::GetOrCreateInstance()->getIOCollection(); } diff --git a/src/simplnx/Utilities/DataStoreUtilities.hpp b/src/simplnx/Utilities/DataStoreUtilities.hpp index a9072bf44d..5a4014c4a8 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.hpp +++ b/src/simplnx/Utilities/DataStoreUtilities.hpp @@ -13,10 +13,16 @@ namespace nx::core::DataStoreUtilities { /** - * @brief Returns the application's DataIOCollection. - * @return + * @brief Returns a non-owning reference to the application's DataIOCollection. + * + * The DataIOCollection is owned by the Application singleton and lives for the + * entire process lifetime. Callers receive a reference, not a shared_ptr, to + * make the non-ownership relationship explicit and prevent accidental lifetime + * extension. + * + * @return Reference to the Application's DataIOCollection. */ -SIMPLNX_EXPORT std::shared_ptr GetIOCollection(); +SIMPLNX_EXPORT DataIOCollection& GetIOCollection(); template uint64 CalculateDataSize(const ShapeType& tupleShape, const ShapeType& componentShape) @@ -36,13 +42,17 @@ uint64 CalculateDataSize(const ShapeType& tupleShape, const ShapeType& component * In Preflight mode, returns an EmptyDataStore that records shape metadata * without allocating any storage. In Execute mode, forwards directly to * createDataStoreWithType() which creates either an in-memory DataStore - * (for "" or k_InMemoryFormat) or an OOC-backed store (for "HDF5-OOC" etc.). + * (for "" unset or k_InMemoryFormat explicit) or an OOC-backed store + * (for "HDF5-OOC" etc.). * * @tparam T Primitive type (int8, float32, uint64, etc.) * @param tupleShape The tuple dimensions (e.g., {100, 200, 300} for a 3D volume) * @param componentShape The component dimensions (e.g., {3} for a 3-component vector) * @param mode PREFLIGHT returns an EmptyDataStore; EXECUTE allocates real storage - * @param dataFormat The already-resolved format name. Empty string means in-memory. + * @param dataFormat The already-resolved format name. An empty string means + * "unset/auto — default to in-memory". k_InMemoryFormat means + * "explicit in-memory". Any other non-empty value must be a + * plugin-registered format name (e.g., "HDF5-OOC"). * @return Shared pointer to the created AbstractDataStore */ template @@ -54,8 +64,7 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap return std::make_unique>(tupleShape, componentShape, dataFormat); } case IDataAction::Mode::Execute: { - auto ioCollection = GetIOCollection(); - return ioCollection->createDataStoreWithType(dataFormat, tupleShape, componentShape); + return GetIOCollection().createDataStoreWithType(dataFormat, tupleShape, componentShape); } default: { throw std::runtime_error("Invalid mode"); @@ -73,12 +82,16 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap * In Preflight mode, returns an EmptyListStore that records shape metadata * without allocating any storage. In Execute mode, forwards directly to * createListStoreWithType() which creates either an in-memory ListStore - * (for "" or k_InMemoryFormat) or an OOC-backed store (for "HDF5-OOC" etc.). + * (for "" unset or k_InMemoryFormat explicit) or an OOC-backed store + * (for "HDF5-OOC" etc.). * * @tparam T Primitive type of the list elements * @param tupleShape The tuple dimensions * @param mode PREFLIGHT returns an EmptyListStore; EXECUTE allocates real storage - * @param dataFormat The already-resolved format name. Empty string means in-memory. + * @param dataFormat The already-resolved format name. An empty string means + * "unset/auto — default to in-memory". k_InMemoryFormat means + * "explicit in-memory". Any other non-empty value must be a + * plugin-registered format name (e.g., "HDF5-OOC"). * @return Shared pointer to the created AbstractListStore */ template @@ -91,8 +104,7 @@ std::shared_ptr> CreateListStore(const ShapeType& tupleShap return std::make_unique>(tupleShape); } case IDataAction::Mode::Execute: { - auto ioCollection = GetIOCollection(); - return ioCollection->createListStoreWithType(dataFormat, tupleShape); + return GetIOCollection().createListStoreWithType(dataFormat, tupleShape); } default: { throw std::runtime_error("Invalid mode"); @@ -108,8 +120,7 @@ std::shared_ptr> ConvertDataStore(const AbstractDataStore> newStore = ioCollection->createDataStoreWithType(dataFormat, dataStore.getTupleShape(), dataStore.getComponentShape()); + std::shared_ptr> newStore = GetIOCollection().createDataStoreWithType(dataFormat, dataStore.getTupleShape(), dataStore.getComponentShape()); if(newStore == nullptr) { return nullptr; diff --git a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp index af84d03e72..ece5cc6ae8 100644 --- a/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp +++ b/src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.cpp @@ -2369,7 +2369,7 @@ Result<> DREAM3D::WriteRecoveryFile(const fs::path& path, const DataStructure& d // Obtain the global DataIOCollection so we can activate the write-array-override. // The SimplnxOoc plugin registers a callback on this collection at startup that // knows how to write OOC array placeholders instead of full data. - auto ioCollection = DataStoreUtilities::GetIOCollection(); + auto& ioCollection = DataStoreUtilities::GetIOCollection(); // The RAII guard sets the override active on construction. While active, // HDF5::DataStructureWriter will check each DataArray against the override @@ -2626,8 +2626,7 @@ Result LoadDataStructureWithHandler(const fs::path& filePath, con auto dataFileReader = nx::core::HDF5::FileIO::ReadFile(filePath); // Check if a handler is registered (e.g. OOC plugin) - auto ioCollection = DataStoreUtilities::GetIOCollection(); - const bool useDeferredLoad = ioCollection->hasDataStoreImportHandler(); + const bool useDeferredLoad = DataStoreUtilities::GetIOCollection().hasDataStoreImportHandler(); // Expand to include ancestor containers, sorted shortest-first auto allPaths = DREAM3D::ExpandSelectedPathsToAncestors(paths); @@ -2651,7 +2650,7 @@ Result LoadDataStructureWithHandler(const fs::path& filePath, con if(useDeferredLoad) { auto eagerLoad = [&dataFileReader](DataStructure& ds, const DataPath& path) -> Result<> { return EagerLoadDataFromHDF5(ds, path, dataFileReader); }; - auto handlerResult = ioCollection->runDataStoreImportHandler(dataStructure, paths, dataFileReader, eagerLoad); + auto handlerResult = DataStoreUtilities::GetIOCollection().runDataStoreImportHandler(dataStructure, paths, dataFileReader, eagerLoad); if(handlerResult.invalid()) { return ConvertInvalidResult(std::move(handlerResult)); diff --git a/test/AppTest.cpp b/test/AppTest.cpp index 434c4fb63e..6cc841fff4 100644 --- a/test/AppTest.cpp +++ b/test/AppTest.cpp @@ -276,12 +276,6 @@ TEST_CASE("Application::getIOCollection", "[Application]") { auto app = Application::GetOrCreateInstance(); - SECTION("IOCollection is never null") - { - auto collection = app->getIOCollection(); - REQUIRE(collection != nullptr); - } - SECTION("getDataStoreFormats returns format names") { auto formats = app->getDataStoreFormats(); diff --git a/test/IOFormat.cpp b/test/IOFormat.cpp index 2b54479c66..4263f1f364 100644 --- a/test/IOFormat.cpp +++ b/test/IOFormat.cpp @@ -13,8 +13,8 @@ TEST_CASE("Contains HDF5 IO Support", "IOTest") { auto app = Application::GetOrCreateInstance(); - auto ioCollection = app->getIOCollection(); - auto h5IO = ioCollection->getManager("HDF5"); + auto& ioCollection = app->getIOCollection(); + auto h5IO = ioCollection.getManager("HDF5"); REQUIRE(h5IO != nullptr); } @@ -101,15 +101,15 @@ TEST_CASE("Data Format: Cannot register IO manager with reserved InMemory name", } }; - auto ioCollection = Application::GetOrCreateInstance()->getIOCollection(); + auto& ioCollection = Application::GetOrCreateInstance()->getIOCollection(); auto badManager = std::make_shared(); - REQUIRE_THROWS(ioCollection->addIOManager(badManager)); + REQUIRE_THROWS(ioCollection.addIOManager(badManager)); } TEST_CASE("Data Format: resolveFormat returns empty with no resolver registered", "[IOTest][DataFormat]") { auto* prefs = Application::GetOrCreateInstance()->getPreferences(); - auto ioCollection = Application::GetOrCreateInstance()->getIOCollection(); + auto& ioCollection = Application::GetOrCreateInstance()->getIOCollection(); std::string savedFormat = prefs->largeDataFormat(); @@ -120,7 +120,7 @@ TEST_CASE("Data Format: resolveFormat returns empty with no resolver registered" DataStructure ds; DataPath dp({"TestArray"}); uint64 largeSize = 2ULL * 1024 * 1024 * 1024; // 2 GB — well above any threshold - std::string dataFormat = ioCollection->resolveFormat(ds, dp, DataType::float32, largeSize); + std::string dataFormat = ioCollection.resolveFormat(ds, dp, DataType::float32, largeSize); REQUIRE(dataFormat.empty()); // Should NOT be changed to any OOC format prefs->setLargeDataFormat(savedFormat); @@ -129,7 +129,7 @@ TEST_CASE("Data Format: resolveFormat returns empty with no resolver registered" TEST_CASE("Data Format: resolveFormat returns empty when format not configured", "[IOTest][DataFormat]") { auto* prefs = Application::GetOrCreateInstance()->getPreferences(); - auto ioCollection = Application::GetOrCreateInstance()->getIOCollection(); + auto& ioCollection = Application::GetOrCreateInstance()->getIOCollection(); std::string savedFormat = prefs->largeDataFormat(); @@ -139,7 +139,7 @@ TEST_CASE("Data Format: resolveFormat returns empty when format not configured", DataStructure ds; DataPath dp({"TestArray"}); uint64 largeSize = 2ULL * 1024 * 1024 * 1024; - std::string dataFormat = ioCollection->resolveFormat(ds, dp, DataType::float32, largeSize); + std::string dataFormat = ioCollection.resolveFormat(ds, dp, DataType::float32, largeSize); REQUIRE(dataFormat.empty()); // No OOC format available prefs->setLargeDataFormat(savedFormat); From 404284caff0104adc4169694c57d56b6d6868a79 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 14 Apr 2026 10:54:05 -0400 Subject: [PATCH 09/13] REFACTOR: Replace runtime_error throws in bulk I/O APIs with Result<> Convert DataIOCollection::addIOManager and the AbstractDataStore copyIntoBuffer/copyFromBuffer family from throwing std::runtime_error / std::out_of_range to returning Result<>. Updates all call sites to propagate errors: Application::loadPlugin, Create{Vertex,1D,2D,3D}Geometry actions, WriteVtkRectilinearGrid, and FillBadData phases (early-return on error). IOFormat test now uses SIMPLNX_RESULT_REQUIRE_INVALID. --- .../Filters/Algorithms/FillBadData.cpp | 50 +++++++++++++++---- .../Filters/Algorithms/FillBadData.hpp | 8 +-- .../Algorithms/WriteVtkRectilinearGrid.cpp | 9 +++- .../src/SimplnxCore/utils/VtkUtilities.hpp | 10 +++- src/simplnx/Core/Application.cpp | 7 ++- .../DataStructure/AbstractDataStore.hpp | 16 +++--- src/simplnx/DataStructure/DataStore.hpp | 18 ++++--- src/simplnx/DataStructure/EmptyDataStore.hpp | 30 +++++------ .../IO/Generic/DataIOCollection.cpp | 10 ++-- .../IO/Generic/DataIOCollection.hpp | 4 +- .../Filter/Actions/CreateGeometry1DAction.hpp | 14 +++++- .../Filter/Actions/CreateGeometry2DAction.hpp | 14 +++++- .../Filter/Actions/CreateGeometry3DAction.hpp | 14 +++++- .../Actions/CreateVertexGeometryAction.hpp | 7 ++- test/IOFormat.cpp | 4 +- test/IParallelAlgorithmTest.cpp | 8 +-- 16 files changed, 161 insertions(+), 62 deletions(-) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp index 7c9c0eb7cd..7cf31a305a 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp @@ -300,7 +300,7 @@ const std::atomic_bool& FillBadData::getCancel() const // @param provisionalLabels Map from voxel index to assigned provisional label // @param dims Image dimensions [X, Y, Z] // ============================================================================= -void FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims) +Result<> FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims) { int64 nextLabel = -1; const usize slabSize = static_cast(dims[0]) * static_cast(dims[1]); @@ -312,7 +312,13 @@ void FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwar for(int64 z = 0; z < dims[2]; z++) { const usize slabStart = static_cast(z) * slabSize; - featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(curSlab.data(), slabSize)); + auto readResult = featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(curSlab.data(), slabSize)); + if(readResult.invalid()) + { + return MergeResults(readResult, + MakeErrorResult(-71500, fmt::format("FillBadData phase 1 (connected component labeling): failed to read Z-slab {} (start index {}, size {}) from feature IDs store.", z, + slabStart, slabSize))); + } for(int64 y = 0; y < dims[1]; y++) { @@ -380,6 +386,7 @@ void FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwar std::swap(prevSlab, curSlab); } + return {}; } // ============================================================================= @@ -421,8 +428,8 @@ void FillBadData::phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std:: // @param unionFind Union-Find structure with resolved equivalences (from Phase 2) // @param maxPhase Maximum existing phase value (for new phase assignment) // ============================================================================= -void FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, - const std::unordered_set& /*smallRegions*/, ChunkAwareUnionFind& unionFind, usize maxPhase) const +Result<> FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, + const std::unordered_set& /*smallRegions*/, ChunkAwareUnionFind& unionFind, usize maxPhase) const { // Classify regions by size std::unordered_map rootSizes; @@ -453,7 +460,12 @@ void FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, for(usize z = 0; z < udims[2]; z++) { const usize slabStart = z * slabSize; - featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(slab.data(), slabSize)); + auto readResult = featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(slab.data(), slabSize)); + if(readResult.invalid()) + { + return MergeResults(readResult, MakeErrorResult(-71501, fmt::format("FillBadData phase 3 (region classification): failed to read Z-slab {} (start index {}, size {}) from feature IDs store.", z, + slabStart, slabSize))); + } bool slabModified = false; for(usize localIdx = 0; localIdx < slabSize; localIdx++) @@ -483,9 +495,16 @@ void FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, if(slabModified) { - featureIdsStore.copyFromBuffer(slabStart, nonstd::span(slab.data(), slabSize)); + auto writeResult = featureIdsStore.copyFromBuffer(slabStart, nonstd::span(slab.data(), slabSize)); + if(writeResult.invalid()) + { + return MergeResults(writeResult, + MakeErrorResult(-71502, fmt::format("FillBadData phase 3 (region classification): failed to write Z-slab {} (start index {}, size {}) back to feature IDs store.", z, + slabStart, slabSize))); + } } } + return {}; } // ============================================================================= @@ -684,7 +703,12 @@ Result<> FillBadData::operator()() const for(usize offset = 0; offset < totalPoints; offset += bufSize) { const usize count = std::min(bufSize, totalPoints - offset); - featureIdsStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); + auto readResult = featureIdsStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); + if(readResult.invalid()) + { + return MergeResults(readResult, + MakeErrorResult(-71503, fmt::format("FillBadData: failed to scan feature IDs store for maximum feature id (chunk [{}, {}) of {}).", offset, offset + count, totalPoints))); + } for(usize i = 0; i < count; i++) { if(buf[i] > static_cast(numFeatures)) @@ -700,13 +724,21 @@ Result<> FillBadData::operator()() const std::unordered_set smallRegions; m_MessageHandler({IFilter::Message::Type::Info, "Phase 1/4: Labeling connected components..."}); - phaseOneCCL(featureIdsStore, unionFind, provisionalLabels, dims); + auto phaseOneResult = phaseOneCCL(featureIdsStore, unionFind, provisionalLabels, dims); + if(phaseOneResult.invalid()) + { + return phaseOneResult; + } m_MessageHandler({IFilter::Message::Type::Info, "Phase 2/4: Resolving region equivalences..."}); phaseTwoGlobalResolution(unionFind, smallRegions); m_MessageHandler({IFilter::Message::Type::Info, "Phase 3/4: Classifying region sizes..."}); - phaseThreeRelabeling(featureIdsStore, cellPhasesPtr, provisionalLabels, smallRegions, unionFind, maxPhase); + auto phaseThreeResult = phaseThreeRelabeling(featureIdsStore, cellPhasesPtr, provisionalLabels, smallRegions, unionFind, maxPhase); + if(phaseThreeResult.invalid()) + { + return phaseThreeResult; + } m_MessageHandler({IFilter::Message::Type::Info, "Phase 4/4: Filling small defects..."}); phaseFourIterativeFill(featureIdsStore, dims, numFeatures); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp index 1e994f2948..e40989040e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp @@ -108,8 +108,9 @@ class SIMPLNXCORE_EXPORT FillBadData * @param unionFind Union-find structure for tracking equivalences * @param provisionalLabels Map from voxel index to provisional label * @param dims Image geometry dimensions + * @return Result<> invalid if a bulk read from the feature IDs store fails. */ - static void phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims); + static Result<> phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims); /** * @brief Phase 2: Global resolution of equivalences and region classification @@ -126,9 +127,10 @@ class SIMPLNXCORE_EXPORT FillBadData * @param smallRegions Set of labels for small regions * @param unionFind Union-find for looking up equivalences * @param maxPhase Maximum phase value (for new phase assignment) + * @return Result<> invalid if a bulk read or write against the feature IDs store fails. */ - void phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, - const std::unordered_set& smallRegions, ChunkAwareUnionFind& unionFind, size_t maxPhase) const; + Result<> phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, + const std::unordered_set& smallRegions, ChunkAwareUnionFind& unionFind, size_t maxPhase) const; /** * @brief Phase 4: Iterative morphological fill diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteVtkRectilinearGrid.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteVtkRectilinearGrid.cpp index c2862ad4ab..454566da91 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteVtkRectilinearGrid.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteVtkRectilinearGrid.cpp @@ -69,8 +69,13 @@ Result<> WriteVtkRectilinearGrid::operator()() for(const DataPath& arrayPath : m_InputValues->SelectedDataArrayPaths) { - ExecuteDataFunction(WriteVtkDataArrayFunctor{}, m_DataStructure.getDataAs(arrayPath)->getDataType(), outputFile, m_InputValues->WriteBinaryFile, m_DataStructure, arrayPath, - m_MessageHandler); + auto writeArrayResult = ExecuteDataFunction(WriteVtkDataArrayFunctor{}, m_DataStructure.getDataAs(arrayPath)->getDataType(), outputFile, m_InputValues->WriteBinaryFile, + m_DataStructure, arrayPath, m_MessageHandler); + if(writeArrayResult.invalid()) + { + fclose(outputFile); + return MergeResults(writeArrayResult, MakeErrorResult(-2091, fmt::format("Error writing data array '{}' to VTK file '{}'", arrayPath.toString(), m_InputValues->OutputFile.string()))); + } } fclose(outputFile); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp index 5350e4efab..9d96dfb7ae 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp @@ -141,7 +141,7 @@ std::string TypeForPrimitive(const IFilter::MessageHandler& messageHandler) struct WriteVtkDataArrayFunctor { template - void operator()(FILE* outputFile, bool binary, DataStructure& dataStructure, const DataPath& arrayPath, const IFilter::MessageHandler& messageHandler) + Result<> operator()(FILE* outputFile, bool binary, DataStructure& dataStructure, const DataPath& arrayPath, const IFilter::MessageHandler& messageHandler) { auto* dataArray = dataStructure.getDataAs>(arrayPath); auto& dataStore = dataArray->getDataStoreRef(); @@ -200,7 +200,12 @@ struct WriteVtkDataArrayFunctor { // General case: bulk read into buffer, byte-swap, then write. std::vector buf(count); - dataStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); + auto copyResult = dataStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); + if(copyResult.invalid()) + { + return MakeErrorResult(-2090, fmt::format("Failed to read chunk [{}, {}) from data array '{}' while writing VTK file: {}", offset, offset + count, arrayPath.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } if constexpr(endian::little == endian::native) { // VTK legacy binary requires big-endian. Swap in the local @@ -250,6 +255,7 @@ struct WriteVtkDataArrayFunctor buffer.append("\n"); fprintf(outputFile, "%s", buffer.c_str()); } + return {}; } }; diff --git a/src/simplnx/Core/Application.cpp b/src/simplnx/Core/Application.cpp index 2b19b18de0..04300f2569 100644 --- a/src/simplnx/Core/Application.cpp +++ b/src/simplnx/Core/Application.cpp @@ -390,7 +390,12 @@ Result<> Application::loadPlugin(const std::filesystem::path& path, bool verbose for(const auto& pluginIO : plugin->getDataIOManagers()) { - m_DataIOCollection->addIOManager(pluginIO); + auto addManagerResult = m_DataIOCollection->addIOManager(pluginIO); + if(addManagerResult.invalid()) + { + return MakeErrorResult( + -34, fmt::format("Failed to register data I/O manager from plugin '{}': {}", plugin->getName(), addManagerResult.errors().empty() ? "unknown error" : addManagerResult.errors()[0].message)); + } } return {}; diff --git a/src/simplnx/DataStructure/AbstractDataStore.hpp b/src/simplnx/DataStructure/AbstractDataStore.hpp index 2a4fe11ce0..78a5140210 100644 --- a/src/simplnx/DataStructure/AbstractDataStore.hpp +++ b/src/simplnx/DataStructure/AbstractDataStore.hpp @@ -425,7 +425,7 @@ class AbstractDataStore : public IDataStore * element range into the appropriate chunk reads from the backing HDF5 * file, coalescing I/O where possible. The caller does not need to know * the chunk layout. - * - **Empty (EmptyDataStore):** Throws std::runtime_error because no data + * - **Empty (EmptyDataStore):** Returns an invalid Result<> because no data * exists. * * The number of elements to copy is determined by `buffer.size()`. The caller @@ -435,10 +435,10 @@ class AbstractDataStore : public IDataStore * @param startIndex The starting flat element index to read from * @param buffer A span to receive the copied values; its size determines how * many elements are read - * @throw std::out_of_range If the requested range exceeds the store's size - * @throw std::runtime_error If called on an EmptyDataStore + * @return Result<> valid on success; invalid with an error message if the + * requested range exceeds the store's size or the store has no data. */ - virtual void copyIntoBuffer(usize startIndex, nonstd::span buffer) const = 0; + virtual Result<> copyIntoBuffer(usize startIndex, nonstd::span buffer) const = 0; /** * @brief Copies values from the provided caller-owned buffer into a @@ -453,7 +453,7 @@ class AbstractDataStore : public IDataStore * - **Out-of-core (OOC stores):** The OOC subclass translates the flat * element range into the appropriate chunk writes to the backing HDF5 * file. - * - **Empty (EmptyDataStore):** Throws std::runtime_error because no data + * - **Empty (EmptyDataStore):** Returns an invalid Result<> because no data * exists. * * The number of elements to copy is determined by `buffer.size()`. The caller @@ -463,10 +463,10 @@ class AbstractDataStore : public IDataStore * @param startIndex The starting flat element index to write to * @param buffer A span containing the values to copy into the store; its * size determines how many elements are written - * @throw std::out_of_range If the requested range exceeds the store's size - * @throw std::runtime_error If called on an EmptyDataStore + * @return Result<> valid on success; invalid with an error message if the + * requested range exceeds the store's size or the store has no data. */ - virtual void copyFromBuffer(usize startIndex, nonstd::span buffer) = 0; + virtual Result<> copyFromBuffer(usize startIndex, nonstd::span buffer) = 0; /** * @brief Returns the value found at the specified index of the DataStore. diff --git a/src/simplnx/DataStructure/DataStore.hpp b/src/simplnx/DataStructure/DataStore.hpp index 0b3e44ebfa..e3b6dd9357 100644 --- a/src/simplnx/DataStructure/DataStore.hpp +++ b/src/simplnx/DataStructure/DataStore.hpp @@ -328,20 +328,23 @@ class DataStore : public AbstractDataStore * @param startIndex The starting flat element index to read from * @param buffer A span to receive the copied values; its size determines how * many elements are read - * @throw std::out_of_range If `[startIndex, startIndex + buffer.size())` exceeds getSize() + * @return Result<> valid on success; invalid if `[startIndex, startIndex + buffer.size())` + * exceeds getSize(). */ - void copyIntoBuffer(usize startIndex, nonstd::span buffer) const override + Result<> copyIntoBuffer(usize startIndex, nonstd::span buffer) const override { const usize count = buffer.size(); // Bounds check: ensure the requested range fits within the store if(startIndex + count > this->getSize()) { - throw std::out_of_range(fmt::format("DataStore::copyIntoBuffer: range [{}, {}) exceeds size {}", startIndex, startIndex + count, this->getSize())); + return MakeErrorResult(-6020, fmt::format("DataStore bulk read failed: requested range [{}, {}) exceeds store size ({}). Requested {} elements starting at index {}.", startIndex, + startIndex + count, this->getSize(), count, startIndex)); } // Direct memory copy from the contiguous backing array into the caller's buffer std::copy(m_Data.get() + startIndex, m_Data.get() + startIndex + count, buffer.data()); + return {}; } /** @@ -352,20 +355,23 @@ class DataStore : public AbstractDataStore * @param startIndex The starting flat element index to write to * @param buffer A span containing the values to write; its size determines * how many elements are written - * @throw std::out_of_range If `[startIndex, startIndex + buffer.size())` exceeds getSize() + * @return Result<> valid on success; invalid if `[startIndex, startIndex + buffer.size())` + * exceeds getSize(). */ - void copyFromBuffer(usize startIndex, nonstd::span buffer) override + Result<> copyFromBuffer(usize startIndex, nonstd::span buffer) override { const usize count = buffer.size(); // Bounds check: ensure the requested range fits within the store if(startIndex + count > this->getSize()) { - throw std::out_of_range(fmt::format("DataStore::copyFromBuffer: range [{}, {}) exceeds size {}", startIndex, startIndex + count, this->getSize())); + return MakeErrorResult(-6021, fmt::format("DataStore bulk write failed: requested range [{}, {}) exceeds store size ({}). Requested {} elements starting at index {}.", startIndex, + startIndex + count, this->getSize(), count, startIndex)); } // Direct memory copy from the caller's buffer into the contiguous backing array std::copy(buffer.begin(), buffer.end(), m_Data.get() + startIndex); + return {}; } /** diff --git a/src/simplnx/DataStructure/EmptyDataStore.hpp b/src/simplnx/DataStructure/EmptyDataStore.hpp index 14f8b4c315..eb8c5c3d9a 100644 --- a/src/simplnx/DataStructure/EmptyDataStore.hpp +++ b/src/simplnx/DataStructure/EmptyDataStore.hpp @@ -189,31 +189,33 @@ class EmptyDataStore : public AbstractDataStore } /** - * @brief Always throws because EmptyDataStore holds no data. EmptyDataStore - * is a metadata-only placeholder used during preflight; bulk data access is - * not supported. The store must be replaced with a real DataStore or OOC - * store before any data I/O is attempted. + * @brief Always returns an invalid Result because EmptyDataStore holds no + * data. EmptyDataStore is a metadata-only placeholder used during preflight; + * bulk data access is not supported. The store must be replaced with a real + * DataStore or OOC store before any data I/O is attempted. * @param startIndex Unused * @param buffer Unused - * @throw std::runtime_error Always + * @return Invalid Result<> — always. */ - void copyIntoBuffer(usize startIndex, nonstd::span buffer) const override + Result<> copyIntoBuffer(usize startIndex, nonstd::span buffer) const override { - throw std::runtime_error("EmptyDataStore::copyIntoBuffer() is not implemented"); + return MakeErrorResult(-6022, "EmptyDataStore bulk read is not supported: EmptyDataStore is a metadata-only placeholder used during preflight and must be replaced with a real DataStore or " + "out-of-core store before bulk I/O is attempted."); } /** - * @brief Always throws because EmptyDataStore holds no data. EmptyDataStore - * is a metadata-only placeholder used during preflight; bulk data access is - * not supported. The store must be replaced with a real DataStore or OOC - * store before any data I/O is attempted. + * @brief Always returns an invalid Result because EmptyDataStore holds no + * data. EmptyDataStore is a metadata-only placeholder used during preflight; + * bulk data access is not supported. The store must be replaced with a real + * DataStore or OOC store before any data I/O is attempted. * @param startIndex Unused * @param buffer Unused - * @throw std::runtime_error Always + * @return Invalid Result<> — always. */ - void copyFromBuffer(usize startIndex, nonstd::span buffer) override + Result<> copyFromBuffer(usize startIndex, nonstd::span buffer) override { - throw std::runtime_error("EmptyDataStore::copyFromBuffer() is not implemented"); + return MakeErrorResult(-6023, "EmptyDataStore bulk write is not supported: EmptyDataStore is a metadata-only placeholder used during preflight and must be replaced with a real DataStore or " + "out-of-core store before bulk I/O is attempted."); } /** diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp index c9ec8d8c79..a2fe2a8a59 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.cpp @@ -24,15 +24,16 @@ DataIOCollection::DataIOCollection() auto coreManager = std::make_shared(); m_ManagerMap[coreManager->formatName()] = coreManager; - addIOManager(std::make_shared()); + // HDF5 format name is not reserved, so this cannot fail. + (void)addIOManager(std::make_shared()); } DataIOCollection::~DataIOCollection() noexcept = default; -void DataIOCollection::addIOManager(std::shared_ptr manager) +Result<> DataIOCollection::addIOManager(std::shared_ptr manager) { if(manager == nullptr) { - return; + return MakeErrorResult(-6010, "Cannot register a null IDataIOManager"); } const std::string& name = manager->formatName(); @@ -41,10 +42,11 @@ void DataIOCollection::addIOManager(std::shared_ptr manager) // under this name (e.g., from a plugin) is rejected. if(name == Preferences::k_InMemoryFormat) { - throw std::runtime_error(fmt::format("Cannot register an I/O manager with the reserved format name '{}'", std::string(Preferences::k_InMemoryFormat))); + return MakeErrorResult(-6011, fmt::format("Cannot register an I/O manager with the reserved format name '{}'", std::string(Preferences::k_InMemoryFormat))); } m_ManagerMap[name] = manager; + return {}; } std::shared_ptr DataIOCollection::getManager(const std::string& formatName) const diff --git a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp index 82c985b9e7..2a3d92c870 100644 --- a/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp +++ b/src/simplnx/DataStructure/IO/Generic/DataIOCollection.hpp @@ -273,8 +273,10 @@ class SIMPLNX_EXPORT DataIOCollection /** * Adds a specified data IO manager for reading and writing to the target format. * @param manager + * @return Result<> with an error if the manager is null or attempts to register + * under the reserved k_InMemoryFormat name. */ - void addIOManager(std::shared_ptr manager); + Result<> addIOManager(std::shared_ptr manager); /** * @brief Returns the IDataIOManager for the specified format name. diff --git a/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp b/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp index cca51870d8..1a4221a0ff 100644 --- a/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp +++ b/src/simplnx/Filter/Actions/CreateGeometry1DAction.hpp @@ -153,13 +153,23 @@ class CreateGeometry1DAction : public IDataCreationAction if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(vertexTupleShape, ShapeType{3}, std::optional{}); - vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-5410, fmt::format("{}Failed to materialize out-of-core vertices array '{}' into in-core store: {}", prefix, m_InputVertices.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } vertices->setDataStore(std::move(inCoreStore)); } if(edges->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(edgeTupleShape, ShapeType{2}, std::optional{}); - edges->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = edges->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-5411, fmt::format("{}Failed to materialize out-of-core edges array '{}' into in-core store: {}", prefix, m_InputEdges.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } edges->setDataStore(std::move(inCoreStore)); } } diff --git a/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp b/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp index 58aa0495e2..fe8ae75127 100644 --- a/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp +++ b/src/simplnx/Filter/Actions/CreateGeometry2DAction.hpp @@ -153,13 +153,23 @@ class CreateGeometry2DAction : public IDataCreationAction if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(vertexTupleShape, ShapeType{3}, std::optional{}); - vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-5510, fmt::format("{}Failed to materialize out-of-core vertices array '{}' into in-core store: {}", prefix, m_InputVertices.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } vertices->setDataStore(std::move(inCoreStore)); } if(faces->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(faceTupleShape, ShapeType{Geometry2DType::k_NumVerts}, std::optional{}); - faces->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = faces->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-5511, fmt::format("{}Failed to materialize out-of-core faces array '{}' into in-core store: {}", prefix, m_InputFaces.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } faces->setDataStore(std::move(inCoreStore)); } } diff --git a/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp b/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp index b60c4a96ed..5b32659ab1 100644 --- a/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp +++ b/src/simplnx/Filter/Actions/CreateGeometry3DAction.hpp @@ -153,13 +153,23 @@ class CreateGeometry3DAction : public IDataCreationAction if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(vertexTupleShape, ShapeType{3}, std::optional{}); - vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-5610, fmt::format("{}Failed to materialize out-of-core vertices array '{}' into in-core store: {}", prefix, m_InputVertices.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } vertices->setDataStore(std::move(inCoreStore)); } if(cells->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(cellTupleShape, ShapeType{Geometry3DType::k_NumVerts}, std::optional{}); - cells->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = cells->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-5611, fmt::format("{}Failed to materialize out-of-core cells array '{}' into in-core store: {}", prefix, m_InputCells.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } cells->setDataStore(std::move(inCoreStore)); } } diff --git a/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp b/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp index 372857bdf5..4a99b2d602 100644 --- a/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp +++ b/src/simplnx/Filter/Actions/CreateVertexGeometryAction.hpp @@ -124,7 +124,12 @@ class CreateVertexGeometryAction : public IDataCreationAction if(vertices->getIDataStore()->getStoreType() == IDataStore::StoreType::OutOfCore) { auto inCoreStore = std::make_shared>(tupleShape, ShapeType{3}, std::optional{}); - vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + auto copyResult = vertices->getDataStoreRef().copyIntoBuffer(0, nonstd::span(inCoreStore->data(), inCoreStore->getSize())); + if(copyResult.invalid()) + { + return MakeErrorResult(-6107, fmt::format("{}Failed to materialize OOC vertices array '{}' into in-core store: {}", prefix, m_InputVertices.toString(), + copyResult.errors().empty() ? "unknown error" : copyResult.errors()[0].message)); + } vertices->setDataStore(std::move(inCoreStore)); } } diff --git a/test/IOFormat.cpp b/test/IOFormat.cpp index 4263f1f364..ae05336b6b 100644 --- a/test/IOFormat.cpp +++ b/test/IOFormat.cpp @@ -4,6 +4,7 @@ #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/DataStructure/IO/Generic/DataIOCollection.hpp" #include "simplnx/DataStructure/IO/Generic/IDataIOManager.hpp" +#include "simplnx/UnitTest/UnitTestCommon.hpp" #include "simplnx/Utilities/DataStoreUtilities.hpp" #include "simplnx/Utilities/MemoryUtilities.hpp" @@ -103,7 +104,8 @@ TEST_CASE("Data Format: Cannot register IO manager with reserved InMemory name", auto& ioCollection = Application::GetOrCreateInstance()->getIOCollection(); auto badManager = std::make_shared(); - REQUIRE_THROWS(ioCollection.addIOManager(badManager)); + auto addResult = ioCollection.addIOManager(badManager); + SIMPLNX_RESULT_REQUIRE_INVALID(addResult); } TEST_CASE("Data Format: resolveFormat returns empty with no resolver registered", "[IOTest][DataFormat]") diff --git a/test/IParallelAlgorithmTest.cpp b/test/IParallelAlgorithmTest.cpp index 8d99291049..2b654a4625 100644 --- a/test/IParallelAlgorithmTest.cpp +++ b/test/IParallelAlgorithmTest.cpp @@ -86,14 +86,14 @@ class MockOocDataStore : public AbstractDataStore throw std::runtime_error("MockOocDataStore::setValue not implemented"); } - void copyIntoBuffer(usize /*startIndex*/, nonstd::span /*buffer*/) const override + Result<> copyIntoBuffer(usize /*startIndex*/, nonstd::span /*buffer*/) const override { - throw std::runtime_error("MockOocDataStore::copyIntoBuffer not implemented"); + return MakeErrorResult(-9001, "MockOocDataStore::copyIntoBuffer not implemented"); } - void copyFromBuffer(usize /*startIndex*/, nonstd::span /*buffer*/) override + Result<> copyFromBuffer(usize /*startIndex*/, nonstd::span /*buffer*/) override { - throw std::runtime_error("MockOocDataStore::copyFromBuffer not implemented"); + return MakeErrorResult(-9002, "MockOocDataStore::copyFromBuffer not implemented"); } value_type at(usize /*index*/) const override From e5d5f8d9f94caa029cb2179ce9fc1a3a9ff5f0e0 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 31 Mar 2026 15:31:49 -0400 Subject: [PATCH 10/13] REFACTOR: Rename algorithm files for OOC dispatch preparation Rename 13 algorithm files to their in-core variant names in preparation for adding OOC (out-of-core) dispatch alternatives. This enables git rename tracking so that subsequent optimization commits show proper diffs against the original algorithm code. Renames (SimplnxCore): FillBadData -> FillBadDataBFS IdentifySample -> IdentifySampleBFS ComputeBoundaryCells -> ComputeBoundaryCellsDirect ComputeFeatureNeighbors -> ComputeFeatureNeighborsDirect ComputeSurfaceAreaToVolume -> ComputeSurfaceAreaToVolumeDirect ComputeSurfaceFeatures -> ComputeSurfaceFeaturesDirect SurfaceNets -> SurfaceNetsDirect QuickSurfaceMesh -> QuickSurfaceMeshDirect DBSCAN -> DBSCANDirect ComputeKMedoids -> ComputeKMedoidsDirect MultiThresholdObjects -> MultiThresholdObjectsDirect Renames (OrientationAnalysis): BadDataNeighborOrientationCheck -> BadDataNeighborOrientationCheckWorklist No logic changes. InputValues structs and filter classes unchanged. --- ...dDataNeighborOrientationCheckWorklist.cpp} | 10 +- ...dDataNeighborOrientationCheckWorklist.hpp} | 14 +- .../BadDataNeighborOrientationCheckFilter.cpp | 4 +- src/Plugins/SimplnxCore/CMakeLists.txt | 22 +- ...lls.cpp => ComputeBoundaryCellsDirect.cpp} | 10 +- ...lls.hpp => ComputeBoundaryCellsDirect.hpp} | 14 +- .../Algorithms/ComputeFeatureNeighbors.cpp | 612 ------------------ .../Algorithms/ComputeFeatureNeighbors.hpp | 56 -- .../ComputeFeatureNeighborsDirect.cpp | 223 +++++++ .../ComputeFeatureNeighborsDirect.hpp | 73 +++ ...KMedoids.cpp => ComputeKMedoidsDirect.cpp} | 16 +- ...KMedoids.hpp => ComputeKMedoidsDirect.hpp} | 14 +- ...p => ComputeSurfaceAreaToVolumeDirect.cpp} | 10 +- ...p => ComputeSurfaceAreaToVolumeDirect.hpp} | 14 +- ...s.cpp => ComputeSurfaceFeaturesDirect.cpp} | 8 +- ...s.hpp => ComputeSurfaceFeaturesDirect.hpp} | 16 +- .../{DBSCAN.cpp => DBSCANDirect.cpp} | 24 +- .../{DBSCAN.hpp => DBSCANDirect.hpp} | 14 +- .../{FillBadData.cpp => FillBadDataBFS.cpp} | 36 +- .../{FillBadData.hpp => FillBadDataBFS.hpp} | 16 +- ...entifySample.cpp => IdentifySampleBFS.cpp} | 18 +- ...entifySample.hpp => IdentifySampleBFS.hpp} | 16 +- ...ts.cpp => MultiThresholdObjectsDirect.cpp} | 8 +- ...ts.hpp => MultiThresholdObjectsDirect.hpp} | 18 +- ...aceMesh.cpp => QuickSurfaceMeshDirect.cpp} | 40 +- ...aceMesh.hpp => QuickSurfaceMeshDirect.hpp} | 14 +- ...{SurfaceNets.cpp => SurfaceNetsDirect.cpp} | 10 +- ...{SurfaceNets.hpp => SurfaceNetsDirect.hpp} | 14 +- .../Filters/ComputeBoundaryCellsFilter.cpp | 4 +- .../Filters/ComputeFeatureNeighborsFilter.cpp | 4 +- .../Filters/ComputeKMedoidsFilter.cpp | 4 +- .../ComputeSurfaceAreaToVolumeFilter.cpp | 4 +- .../Filters/ComputeSurfaceFeaturesFilter.cpp | 4 +- .../src/SimplnxCore/Filters/DBSCANFilter.cpp | 18 +- .../SimplnxCore/Filters/FillBadDataFilter.cpp | 4 +- .../Filters/IdentifySampleFilter.cpp | 4 +- .../Filters/MultiThresholdObjectsFilter.cpp | 4 +- .../Filters/QuickSurfaceMeshFilter.cpp | 4 +- .../SimplnxCore/Filters/SurfaceNetsFilter.cpp | 4 +- src/Plugins/SimplnxCore/test/DBSCANTest.cpp | 34 +- 40 files changed, 532 insertions(+), 904 deletions(-) rename src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/{BadDataNeighborOrientationCheck.cpp => BadDataNeighborOrientationCheckWorklist.cpp} (94%) rename src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/{BadDataNeighborOrientationCheck.hpp => BadDataNeighborOrientationCheckWorklist.hpp} (61%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeBoundaryCells.cpp => ComputeBoundaryCellsDirect.cpp} (88%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeBoundaryCells.hpp => ComputeBoundaryCellsDirect.hpp} (55%) delete mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp delete mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeKMedoids.cpp => ComputeKMedoidsDirect.cpp} (90%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeKMedoids.hpp => ComputeKMedoidsDirect.hpp} (63%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeSurfaceAreaToVolume.cpp => ComputeSurfaceAreaToVolumeDirect.cpp} (93%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeSurfaceAreaToVolume.hpp => ComputeSurfaceAreaToVolumeDirect.hpp} (54%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeSurfaceFeatures.cpp => ComputeSurfaceFeaturesDirect.cpp} (95%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{ComputeSurfaceFeatures.hpp => ComputeSurfaceFeaturesDirect.hpp} (65%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{DBSCAN.cpp => DBSCANDirect.cpp} (97%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{DBSCAN.hpp => DBSCANDirect.hpp} (69%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{FillBadData.cpp => FillBadDataBFS.cpp} (94%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{FillBadData.hpp => FillBadDataBFS.hpp} (89%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{IdentifySample.cpp => IdentifySampleBFS.cpp} (93%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{IdentifySample.hpp => IdentifySampleBFS.hpp} (67%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{MultiThresholdObjects.cpp => MultiThresholdObjectsDirect.cpp} (96%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{MultiThresholdObjects.hpp => MultiThresholdObjectsDirect.hpp} (73%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{QuickSurfaceMesh.cpp => QuickSurfaceMeshDirect.cpp} (97%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{QuickSurfaceMesh.hpp => QuickSurfaceMeshDirect.hpp} (78%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{SurfaceNets.cpp => SurfaceNetsDirect.cpp} (97%) rename src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/{SurfaceNets.hpp => SurfaceNetsDirect.hpp} (70%) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp similarity index 94% rename from src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp rename to src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp index 2937ac640c..1c862c6a5e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp @@ -1,4 +1,4 @@ -#include "BadDataNeighborOrientationCheck.hpp" +#include "BadDataNeighborOrientationCheckWorklist.hpp" #include "simplnx/Common/Numbers.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -12,7 +12,7 @@ using namespace nx::core; // ----------------------------------------------------------------------------- -BadDataNeighborOrientationCheck::BadDataNeighborOrientationCheck(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, +BadDataNeighborOrientationCheckWorklist::BadDataNeighborOrientationCheckWorklist(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, BadDataNeighborOrientationCheckInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) @@ -22,16 +22,16 @@ BadDataNeighborOrientationCheck::BadDataNeighborOrientationCheck(DataStructure& } // ----------------------------------------------------------------------------- -BadDataNeighborOrientationCheck::~BadDataNeighborOrientationCheck() noexcept = default; +BadDataNeighborOrientationCheckWorklist::~BadDataNeighborOrientationCheckWorklist() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& BadDataNeighborOrientationCheck::getCancel() +const std::atomic_bool& BadDataNeighborOrientationCheckWorklist::getCancel() { return m_ShouldCancel; } // ----------------------------------------------------------------------------- -Result<> BadDataNeighborOrientationCheck::operator()() +Result<> BadDataNeighborOrientationCheckWorklist::operator()() { const float misorientationTolerance = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp similarity index 61% rename from src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp rename to src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp index 5a65acab3d..1a2cd42b4e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp @@ -23,17 +23,17 @@ struct ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckInputValues /** * @class */ -class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheck +class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckWorklist { public: - BadDataNeighborOrientationCheck(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + BadDataNeighborOrientationCheckWorklist(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, BadDataNeighborOrientationCheckInputValues* inputValues); - ~BadDataNeighborOrientationCheck() noexcept; + ~BadDataNeighborOrientationCheckWorklist() noexcept; - BadDataNeighborOrientationCheck(const BadDataNeighborOrientationCheck&) = delete; - BadDataNeighborOrientationCheck(BadDataNeighborOrientationCheck&&) noexcept = delete; - BadDataNeighborOrientationCheck& operator=(const BadDataNeighborOrientationCheck&) = delete; - BadDataNeighborOrientationCheck& operator=(BadDataNeighborOrientationCheck&&) noexcept = delete; + BadDataNeighborOrientationCheckWorklist(const BadDataNeighborOrientationCheckWorklist&) = delete; + BadDataNeighborOrientationCheckWorklist(BadDataNeighborOrientationCheckWorklist&&) noexcept = delete; + BadDataNeighborOrientationCheckWorklist& operator=(const BadDataNeighborOrientationCheckWorklist&) = delete; + BadDataNeighborOrientationCheckWorklist& operator=(BadDataNeighborOrientationCheckWorklist&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp index 5f817313bb..3b86e6a7ae 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp @@ -1,5 +1,5 @@ #include "BadDataNeighborOrientationCheckFilter.hpp" -#include "OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp" +#include "OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -135,7 +135,7 @@ Result<> BadDataNeighborOrientationCheckFilter::executeImpl(DataStructure& dataS inputValues.CellPhasesArrayPath = filterArgs.value(k_CellPhasesArrayPath_Key); inputValues.CrystalStructuresArrayPath = filterArgs.value(k_CrystalStructuresArrayPath_Key); - return BadDataNeighborOrientationCheck(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return BadDataNeighborOrientationCheckWorklist(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/CMakeLists.txt b/src/Plugins/SimplnxCore/CMakeLists.txt index a19eff9ce3..30d100f871 100644 --- a/src/Plugins/SimplnxCore/CMakeLists.txt +++ b/src/Plugins/SimplnxCore/CMakeLists.txt @@ -185,7 +185,7 @@ set(AlgorithmList ComputeArrayHistogramByFeature ComputeArrayStatistics ComputeBiasedFeatures - ComputeBoundaryCells + ComputeBoundaryCellsDirect ComputeBoundingBoxStats ComputeCoordinatesImageGeom ComputeCoordinateThreshold @@ -195,21 +195,21 @@ set(AlgorithmList ComputeFeatureBounds ComputeFeatureCentroids ComputeFeatureClustering - ComputeFeatureNeighbors + ComputeFeatureNeighborsDirect ComputeFeaturePhases ComputeFeaturePhasesBinary ComputeFeatureRect ComputeFeatureSizes ComputeGroupingDensity ComputeKMeans - ComputeKMedoids + ComputeKMedoidsDirect ComputeLargestCrossSections ComputeMomentInvariants2D ComputeNeighborhoods ComputeNeighborListStatistics # ComputeNumFeatures - ComputeSurfaceAreaToVolume - ComputeSurfaceFeatures + ComputeSurfaceAreaToVolumeDirect + ComputeSurfaceFeaturesDirect # ComputeTriangleAreas ComputeTriangleGeomCentroids ComputeTriangleGeomVolumes @@ -235,7 +235,7 @@ set(AlgorithmList CropEdgeGeometry CropImageGeometry CropVertexGeometry - DBSCAN + DBSCANDirect # DeleteData ErodeDilateBadData ErodeDilateCoordinationNumber @@ -246,12 +246,12 @@ set(AlgorithmList ExtractPipelineToFile ExtractVertexGeometry FeatureFaceCurvature - FillBadData + FillBadDataBFS FindNRingNeighbors FlyingEdges3D HierarchicalSmooth IdentifyDuplicateVertices - IdentifySample + IdentifySampleBFS InitializeData InitializeImageGeomCellData InterpolatePointCloudToRegularGrid @@ -261,14 +261,14 @@ set(AlgorithmList LaplacianSmoothing MapPointCloudToRegularGrid # MoveData - MultiThresholdObjects + MultiThresholdObjectsDirect NearestPointFuseRegularGrids PadImageGeometry PartitionGeometry PointSampleEdgeGeometry ExtractFeatureBoundaries2D PointSampleTriangleGeometry - QuickSurfaceMesh + QuickSurfaceMeshDirect ReadBinaryCTNorthstar ReadCSVFile ReadDeformKeyFileV12 @@ -303,7 +303,7 @@ set(AlgorithmList SliceTriangleGeometry SplitDataArrayByComponent SplitDataArrayByTuple - SurfaceNets + SurfaceNetsDirect TriangleCentroid TriangleDihedralAngle TriangleNormal diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp similarity index 88% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp index 0c4c602e89..91e50d6aba 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp @@ -1,4 +1,4 @@ -#include "ComputeBoundaryCells.hpp" +#include "ComputeBoundaryCellsDirect.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -7,7 +7,7 @@ using namespace nx::core; // ----------------------------------------------------------------------------- -ComputeBoundaryCells::ComputeBoundaryCells(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues) +ComputeBoundaryCellsDirect::ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -16,16 +16,16 @@ ComputeBoundaryCells::ComputeBoundaryCells(DataStructure& dataStructure, const I } // ----------------------------------------------------------------------------- -ComputeBoundaryCells::~ComputeBoundaryCells() noexcept = default; +ComputeBoundaryCellsDirect::~ComputeBoundaryCellsDirect() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ComputeBoundaryCells::getCancel() +const std::atomic_bool& ComputeBoundaryCellsDirect::getCancel() { return m_ShouldCancel; } // ----------------------------------------------------------------------------- -Result<> ComputeBoundaryCells::operator()() +Result<> ComputeBoundaryCellsDirect::operator()() { const ImageGeom imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const SizeVec3 udims = imageGeometry.getDimensions(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp similarity index 55% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp index 954173dde4..a2e833245b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp @@ -21,16 +21,16 @@ struct SIMPLNXCORE_EXPORT ComputeBoundaryCellsInputValues /** * @class */ -class SIMPLNXCORE_EXPORT ComputeBoundaryCells +class SIMPLNXCORE_EXPORT ComputeBoundaryCellsDirect { public: - ComputeBoundaryCells(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues); - ~ComputeBoundaryCells() noexcept; + ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues); + ~ComputeBoundaryCellsDirect() noexcept; - ComputeBoundaryCells(const ComputeBoundaryCells&) = delete; - ComputeBoundaryCells(ComputeBoundaryCells&&) noexcept = delete; - ComputeBoundaryCells& operator=(const ComputeBoundaryCells&) = delete; - ComputeBoundaryCells& operator=(ComputeBoundaryCells&&) noexcept = delete; + ComputeBoundaryCellsDirect(const ComputeBoundaryCellsDirect&) = delete; + ComputeBoundaryCellsDirect(ComputeBoundaryCellsDirect&&) noexcept = delete; + ComputeBoundaryCellsDirect& operator=(const ComputeBoundaryCellsDirect&) = delete; + ComputeBoundaryCellsDirect& operator=(ComputeBoundaryCellsDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp deleted file mode 100644 index 586286497f..0000000000 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp +++ /dev/null @@ -1,612 +0,0 @@ -#include "ComputeFeatureNeighbors.hpp" - -#include "simplnx/DataStructure/DataArray.hpp" -#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" -#include "simplnx/DataStructure/NeighborList.hpp" -#include "simplnx/Utilities/MessageHelper.hpp" -#include "simplnx/Utilities/NeighborUtilities.hpp" - -using namespace nx::core; - -namespace -{ -template -struct ImageDimensionState -{ - static constexpr bool HasEmptyXDim = EmptyXV; - static constexpr bool HasEmptyYDim = EmptyYV; - static constexpr bool HasEmptyZDim = EmptyZV; - - static constexpr bool Is1DImageDimsState() - { - return (HasEmptyXDim == true && HasEmptyYDim == true && HasEmptyZDim == false) || (HasEmptyXDim == true && HasEmptyYDim == false && HasEmptyZDim == true) || - (HasEmptyXDim == false && HasEmptyYDim == true && HasEmptyZDim == true); - } - - static constexpr bool Is2DImageDimsState() - { - return (HasEmptyXDim == true && HasEmptyYDim == false && HasEmptyZDim == false) || (HasEmptyXDim == false && HasEmptyYDim == true && HasEmptyZDim == false) || - (HasEmptyXDim == false && HasEmptyYDim == false && HasEmptyZDim == true); - } -}; - -using Image3D = ImageDimensionState; -using EmptyXImage2D = ImageDimensionState; -using EmptyYImage2D = ImageDimensionState; -using EmptyZImage2D = ImageDimensionState; -using XImage1D = ImageDimensionState; -using YImage1D = ImageDimensionState; -using ZImage1D = ImageDimensionState; -using SingleVoxelImage = ImageDimensionState; - -template -constexpr bool IsExpectedImageDimsState() -{ - return ActualT::HasEmptyXDim == ExpectedT::HasEmptyXDim && ActualT::HasEmptyYDim == ExpectedT::HasEmptyYDim && ActualT::HasEmptyZDim == ExpectedT::HasEmptyZDim; -} - -template -struct ComputeFeatureNeighborsFunctor -{ - template - Result<> operator()(BoolAbstractDataStore* surfaceFeatures, Int8AbstractDataStore* boundaryCells, Float32NeighborList& sharedSurfaceAreaList, Int32NeighborList& neighborsList, - Int32AbstractDataStore& numNeighbors, const Int32AbstractDataStore& featureIds, usize totalFeatures, const std::array& dims, const std::array spacing, - const std::array& neighborVoxelIndexOffsets, ThrottledMessenger& throttledMessenger, const std::atomic_bool& shouldCancel) const - { - if(ProcessSurfaceFeaturesV) - { - if(surfaceFeatures == nullptr) - { - return MakeErrorResult(-789620, "Process Surface Features selected, but the supplied Surface Features Array invalid."); - } - } - - if(ProcessBoundaryCellsV) - { - if(boundaryCells == nullptr) - { - return MakeErrorResult(-789621, "Process Boundary Cells selected, but the supplied Boundary Cells Array invalid."); - } - } - - const usize totalPoints = featureIds.getNumberOfTuples(); - - const std::array precomputedFaceAreas = computeFaceSurfaceAreas(spacing); - std::vector> neighborSurfaceAreas(totalFeatures); - - /** - * Stage 1: Process Boundary Cells - * - * The primary goal of Stage 1 is to isolate border cell specific checks out of Phase 2 - * (the internal cells). This includes flagging the border cells without needing to - * branch, and ignoring invalid voxel faces inherently as much as possible. This segmentation - * also allows for removing a branch in the deepest nested loop in Phase 2. - * - * Stage 1 has been split into 3 parts, the vertex (corner), edge, and face cells. - * Of these parts there are two main logic flows defined by `processFrameCell` and - * `processFaceCell`, the main difference between the two being that the "Frame" algorithm - * checks every face neighbor and validates them, whereas the "Face" algorithm removes the - * validation check and cuts down the checked faces to only the valid ones. It should also be - * noted that, optimization is being left on the table with the frame section. It could be further - * broken down into processing each edge/voxel individually to mirror the optimization done to - * faces, but the segmentation done here would make it far less readable and in the greater context - * the speed gain is minimal considering they are O(n-2) and O(1) respectively and the greater algorithm - * is 0(6(n-2)^3). - * - * Note here that discussions were had of adding Kahan Summation for calculating the surface - * areas, but was decided against to conserve memory. At least until the issue presents itself - * in a real world dataset. - */ - const std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - const auto processFrameCell = [&](const int64 zIndex, const int64 yIndex, const int64 xIndex) -> void { - int8 numDiffNeighbors = 0; - - const int64 voxelIndex = (dims[0] * dims[1] * zIndex) + (dims[0] * yIndex) + xIndex; - const int32 feature = featureIds.getValue(voxelIndex); - if(feature > 0) - { - if constexpr(ProcessSurfaceFeaturesV && !ImageDimensionStateT::Is1DImageDimsState()) - { - surfaceFeatures->setValue(feature, true); - } - - // Loop over the 6 face neighbors of the voxel - std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIndex, yIndex, zIndex, dims); - for(const auto faceIndex : faceNeighborInternalIdx) // ref more expensive than trivial copy for scalar types - { - if(!isValidFaceNeighbor[faceIndex]) - { - continue; - } - - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - - const int32 neighborFeatureId = featureIds.getValue(neighborPoint); - if(neighborFeatureId != feature && neighborFeatureId > 0) - { - numDiffNeighbors++; - neighborSurfaceAreas[feature][neighborFeatureId] += precomputedFaceAreas[faceIndex]; - } - } - } - if constexpr(ProcessBoundaryCellsV) - { - boundaryCells->setValue(voxelIndex, numDiffNeighbors); - } - }; - - // Process Corners - { - /** - * Process Corners: - * The constexpr logic in the code block will handle the following, using XYZ indexes: - * - * Case 1: Empty X - * - 0,0,0 - * - 0,n_Y,0 - * - 0,0,n_Z - * - 0,n_Y,n_Z - * - * Case 2: Empty Y - * - 0,0,0 - * - n_X,0,0 - * - 0,0,n_Z - * - n_X,0,n_Z - * - * Case 3: Empty Z - * - 0,0,0 - * - n_X,0,0 - * - 0,n_Y,0 - * - n_X,n_Y,0 - * - * Case 4: 3D Image (Image Stack) - * - 0,0,0 - * - n_X,0,0 - * - 0,n_Y,0 - * - 0,0,n_Z - * - n_X,n_Y,0 - * - n_X,0,n_Z - * - 0,n_Y,n_Z - * - n_X,n_Y,n_Z - */ - - processFrameCell(0, 0, 0); - if constexpr(ProcessSurfaceFeaturesV && ImageDimensionStateT::Is1DImageDimsState()) - { - // Since the frame cell function is shared between corners and edges - // 1D case for border feature flagging must be disabled to prevent - // the entire row from being flagged, thus we must do the corners - // in an explicit action. - // Note here that there is an argument that a new function - // should be defined to handle corner cells. However, this was - // decided against to avoid needles code duplication, as the - // difference between the two functions would be a single - // constexpr if statement - const int32 feature = featureIds.getValue(0); - surfaceFeatures->setValue(feature, true); - } - if constexpr(!IsExpectedImageDimsState()) - { - processFrameCell(dims[2] - 1, dims[1] - 1, dims[0] - 1); // If 2D the dims in empty dimension is 1 so this line effectively preforms for all cases - - if constexpr(ProcessSurfaceFeaturesV && ImageDimensionStateT::Is1DImageDimsState()) - { - // Since the frame cell function is shared between corners and edges - // 1D case for border feature flagging must be disabled to prevent - // the entire row from being flagged, thus we must do the corners - // in an explicit action. - // Note here that there is an argument that a new function - // should be defined to handle corner cells. However, this was - // decided against to avoid needles code duplication, as the - // difference between the two functions would be a single - // constexpr if statement - const int64 voxelIndex = (dims[0] * dims[1] * (dims[2] - 1)) + (dims[0] * (dims[1] - 1)) + (dims[0] - 1); - const int32 feature = featureIds.getValue(voxelIndex); - surfaceFeatures->setValue(feature, true); - } - - if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) - { - if constexpr(!IsExpectedImageDimsState()) - { - processFrameCell(0, 0, dims[0] - 1); - } - if constexpr(!IsExpectedImageDimsState()) - { - processFrameCell(0, dims[1] - 1, 0); - } - if constexpr(!IsExpectedImageDimsState()) - { - processFrameCell(dims[2] - 1, 0, 0); - } - if constexpr(IsExpectedImageDimsState()) - { - processFrameCell(0, dims[1] - 1, dims[0] - 1); - processFrameCell(dims[2] - 1, 0, dims[0] - 1); - processFrameCell(dims[2] - 1, dims[1] - 1, 0); - } - } - } - } - - // Case 0: Process Edges - // X Edges - if constexpr((ImageDimensionStateT::Is2DImageDimsState() && !IsExpectedImageDimsState()) || IsExpectedImageDimsState() || - IsExpectedImageDimsState()) - { - for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) - { - processFrameCell(0, 0, xIndex); - if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) - { - if constexpr(IsExpectedImageDimsState()) - { - processFrameCell(0, dims[1] - 1, xIndex); - processFrameCell(dims[2] - 1, 0, xIndex); - } - processFrameCell(dims[2] - 1, dims[1] - 1, xIndex); - } - } - } - - // Y Edges - if constexpr((ImageDimensionStateT::Is2DImageDimsState() && !IsExpectedImageDimsState()) || IsExpectedImageDimsState() || - IsExpectedImageDimsState()) - { - for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) - { - processFrameCell(0, yIndex, 0); - if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) - { - if constexpr(IsExpectedImageDimsState()) - { - processFrameCell(0, yIndex, dims[0] - 1); - processFrameCell(dims[2] - 1, yIndex, 0); - } - processFrameCell(dims[2] - 1, yIndex, dims[0] - 1); - } - } - } - - // Z Edges - if constexpr((ImageDimensionStateT::Is2DImageDimsState() && !IsExpectedImageDimsState()) || IsExpectedImageDimsState() || - IsExpectedImageDimsState()) - { - for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) - { - processFrameCell(zIndex, 0, 0); - if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) - { - if constexpr(IsExpectedImageDimsState()) - { - processFrameCell(zIndex, 0, dims[0] - 1); - processFrameCell(zIndex, dims[1] - 1, 0); - } - processFrameCell(zIndex, dims[1] - 1, dims[0] - 1); - } - } - } - - // Process Planes for 2D and 3D (Stack) Images - if constexpr(!ImageDimensionStateT::Is1DImageDimsState() && !IsExpectedImageDimsState()) - { - const auto processFaceCell = [&](const int64 zIndex, const int64 yIndex, const int64 xIndex, const std::vector& validFaces) -> void { - int8 numDiffNeighbors = 0; - - const int64 voxelIndex = (dims[0] * dims[1] * zIndex) + (dims[0] * yIndex) + xIndex; - const int32 feature = featureIds.getValue(voxelIndex); - if(feature > 0) - { - if constexpr(ProcessSurfaceFeaturesV && IsExpectedImageDimsState()) - { - surfaceFeatures->setValue(feature, true); - } - - // Loop over the face neighbors of the voxel - for(const auto faceIndex : validFaces) // ref more expensive than trivial copy for scalar types - { - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - - const int32 neighborFeatureId = featureIds.getValue(neighborPoint); - if(neighborFeatureId != feature && neighborFeatureId > 0) - { - numDiffNeighbors++; - neighborSurfaceAreas[feature][neighborFeatureId] += precomputedFaceAreas[faceIndex]; - } - } - } - if constexpr(ProcessBoundaryCellsV) - { - boundaryCells->setValue(voxelIndex, numDiffNeighbors); - } - }; - - // Case 1: Z Planes - { - if constexpr(IsExpectedImageDimsState()) - { - std::vector negZValidFaces = {k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; - std::vector posZValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor}; - for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) - { - for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) - { - processFaceCell(0, yIndex, xIndex, negZValidFaces); - processFaceCell(dims[2] - 1, yIndex, xIndex, posZValidFaces); - } - } - } - if constexpr(IsExpectedImageDimsState()) - { - std::vector validFaces = {k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor}; - for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) - { - for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) - { - processFaceCell(0, yIndex, xIndex, validFaces); - } - } - } - } - - // Case 2: Y Planes - { - if constexpr(IsExpectedImageDimsState()) - { - std::vector negYValidFaces = {k_NegativeZNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; - std::vector posYValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveZNeighbor}; - for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) - { - for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) - { - processFaceCell(zIndex, 0, xIndex, negYValidFaces); - processFaceCell(zIndex, dims[1] - 1, xIndex, posYValidFaces); - } - } - } - if constexpr(IsExpectedImageDimsState()) - { - std::vector validFaces = {k_NegativeZNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveZNeighbor}; - for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) - { - for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) - { - processFaceCell(zIndex, 0, xIndex, validFaces); - } - } - } - } - - // Case 3: X Planes - { - if constexpr(IsExpectedImageDimsState()) - { - std::vector negXValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; - std::vector posXValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; - for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) - { - for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) - { - processFaceCell(zIndex, yIndex, 0, negXValidFaces); - processFaceCell(zIndex, yIndex, dims[0] - 1, posXValidFaces); - } - } - } - if constexpr(IsExpectedImageDimsState()) - { - std::vector validFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; - for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) - { - for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) - { - processFaceCell(zIndex, yIndex, 0, validFaces); - } - } - } - } - } - - /** - * Stage 2: Process Internal Cells - * This stage has a bulk of the computation, and runtime branching has been minimized - * to reflect that reality, see comment for Stage 1. This section just walks every - * internal cell and checks each of the neighbors, storing them onto the existing - * results from the boundary cell phases. - */ - if constexpr(IsExpectedImageDimsState()) - { - // Loop over all internal cells to generate the neighbor lists - for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) - { - const int64 zStride = dims[0] * dims[1] * zIndex; - for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) - { - const int64 yStride = dims[0] * yIndex; - throttledMessenger.sendThrottledMessage([&] { return fmt::format("Determining Neighbor Lists || {:.2f}% Complete", CalculatePercentComplete(zStride + yStride, totalPoints)); }); - - if(shouldCancel) - { - return {}; - } - for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) - { - int64 voxelIndex = zStride + yStride + xIndex; - - // This value tracks the number of neighboring cells that have feature ids different from itself - int8 numDiffNeighbors = 0; - int32 feature = featureIds.getValue(voxelIndex); - if(feature > 0) - { - // Loop over the face neighbors of the voxel - for(const auto faceIndex : faceNeighborInternalIdx) // ref more expensive than trivial copy for scalar types - { - // No need for a face validity check because we are only processing internal cells - - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - - const int32 neighborFeatureId = featureIds.getValue(neighborPoint); - if(neighborFeatureId != feature && neighborFeatureId > 0) - { - numDiffNeighbors++; - neighborSurfaceAreas[feature][neighborFeatureId] += precomputedFaceAreas[faceIndex]; - } - } - } - if constexpr(ProcessBoundaryCellsV) - { - boundaryCells->setValue(voxelIndex, numDiffNeighbors); - } - } - } - } - } - - for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) - { - const usize neighborCount = neighborSurfaceAreas[featureIdx].size(); - numNeighbors.setValue(featureIdx, static_cast(neighborCount)); - - // Set the vector for each list into the NeighborList Object - auto sharedNeiLst = std::make_shared::VectorType>(); - sharedNeiLst->reserve(neighborCount); - auto sharedSAL = std::make_shared::VectorType>(); - sharedSAL->reserve(neighborCount); - for(const auto& [featureId, surfaceArea] : neighborSurfaceAreas[featureIdx]) - { - sharedNeiLst->push_back(static_cast(featureId)); - sharedSAL->push_back(static_cast(surfaceArea)); - } - neighborsList.setList(static_cast(featureIdx), sharedNeiLst); - sharedSurfaceAreaList.setList(static_cast(featureIdx), sharedSAL); - } - - return {}; - } -}; - -template -Result<> ProcessVoxels(const FunctorT& functor, const ImageGeom& imageGeom, ArgsT&&... args) -{ - const bool xDimEmpty = imageGeom.getNumXCells() == 1; - const bool yDimEmpty = imageGeom.getNumYCells() == 1; - const bool zDimEmpty = imageGeom.getNumZCells() == 1; - const uint8 emptyDimCount = static_cast(xDimEmpty) + static_cast(yDimEmpty) + static_cast(zDimEmpty); - - // Treat dimensions of 1 as flat for image geom - if(emptyDimCount == 0) - { - return functor.template operator()(std::forward(args)...); - } - if(emptyDimCount == 1) - { - if(zDimEmpty) - { - return functor.template operator()(std::forward(args)...); - } - if(yDimEmpty) - { - return functor.template operator()(std::forward(args)...); - } - if(xDimEmpty) - { - return functor.template operator()(std::forward(args)...); - } - } - if(emptyDimCount == 2) - { - if(xDimEmpty && yDimEmpty) - { - return functor.template operator()(std::forward(args)...); - } - if(xDimEmpty && zDimEmpty) - { - return functor.template operator()(std::forward(args)...); - } - if(yDimEmpty && zDimEmpty) - { - return functor.template operator()(std::forward(args)...); - } - } - if(emptyDimCount == 3) - { - return functor.template operator()(std::forward(args)...); - } - - return {}; -} -} // namespace - -// ----------------------------------------------------------------------------- -ComputeFeatureNeighbors::ComputeFeatureNeighbors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - ComputeFeatureNeighborsInputValues* inputValues) -: m_DataStructure(dataStructure) -, m_InputValues(inputValues) -, m_ShouldCancel(shouldCancel) -, m_MessageHandler(mesgHandler) -{ -} - -// ----------------------------------------------------------------------------- -ComputeFeatureNeighbors::~ComputeFeatureNeighbors() noexcept = default; - -// ----------------------------------------------------------------------------- -Result<> ComputeFeatureNeighbors::operator()() -{ - MessageHelper messageHelper(m_MessageHandler); - ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); - - auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath)->getDataStoreRef(); - auto& numNeighbors = m_DataStructure.getDataAs(m_InputValues->NumberOfNeighborsPath)->getDataStoreRef(); - - auto& neighborsList = m_DataStructure.getDataRefAs(m_InputValues->NeighborListPath); - auto& sharedSurfaceAreaList = m_DataStructure.getDataRefAs(m_InputValues->SharedSurfaceAreaListPath); - - usize totalFeatures = numNeighbors.getNumberOfTuples(); - - /* Ensure that we will be able to work with the user selected featureId Array */ - const int32 maxFeatureId = *std::max_element(featureIds.cbegin(), featureIds.cend()); - if(static_cast(maxFeatureId) >= totalFeatures) - { - std::stringstream out; - out << "Data Array " << m_InputValues->FeatureIdsPath.getTargetName() << " has a maximum value of " << maxFeatureId << " which is greater than the " - << " number of features from array " << m_InputValues->NumberOfNeighborsPath.getTargetName() << " which has " << totalFeatures << ". Did you select the " - << " incorrect array for the 'FeatureIds' array?"; - return MakeErrorResult(-24500, out.str()); - } - - const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometryPath); - SizeVec3 uDims = imageGeom.getDimensions(); - - std::array dims = {static_cast(uDims[0]), static_cast(uDims[1]), static_cast(uDims[2])}; - - FloatVec3 spacing32 = imageGeom.getSpacing(); - - std::array spacing64 = {static_cast(spacing32[0]), static_cast(spacing32[1]), static_cast(spacing32[2])}; - - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - - if(m_InputValues->StoreSurfaceFeatures && m_InputValues->StoreBoundaryCells) - { - // Surface Features filled with `false` by default during creation in preflight - auto* surfaceFeatures = m_DataStructure.getDataAs(m_InputValues->SurfaceFeaturesPath)->getDataStore(); - auto* boundaryCells = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsPath)->getDataStore(); - return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, surfaceFeatures, boundaryCells, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, - spacing64, neighborVoxelIndexOffsets, throttledMessenger, m_ShouldCancel); - } - if(m_InputValues->StoreSurfaceFeatures) - { - // Surface Features filled with `false` by default during creation in preflight - auto* surfaceFeatures = m_DataStructure.getDataAs(m_InputValues->SurfaceFeaturesPath)->getDataStore(); - return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, surfaceFeatures, nullptr, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, - spacing64, neighborVoxelIndexOffsets, throttledMessenger, m_ShouldCancel); - } - if(m_InputValues->StoreBoundaryCells) - { - auto* boundaryCells = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsPath)->getDataStore(); - return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, nullptr, boundaryCells, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, - spacing64, neighborVoxelIndexOffsets, throttledMessenger, m_ShouldCancel); - } - - return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, nullptr, nullptr, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, spacing64, - neighborVoxelIndexOffsets, throttledMessenger, m_ShouldCancel); -} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp deleted file mode 100644 index 810e3a4dd5..0000000000 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp +++ /dev/null @@ -1,56 +0,0 @@ -#pragma once - -#include "SimplnxCore/SimplnxCore_export.hpp" - -#include "simplnx/DataStructure/DataPath.hpp" -#include "simplnx/DataStructure/DataStructure.hpp" -#include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ArraySelectionParameter.hpp" -#include "simplnx/Parameters/AttributeMatrixSelectionParameter.hpp" -#include "simplnx/Parameters/BoolParameter.hpp" -#include "simplnx/Parameters/DataObjectNameParameter.hpp" -#include "simplnx/Parameters/GeometrySelectionParameter.hpp" - -namespace nx::core -{ - -struct SIMPLNXCORE_EXPORT ComputeFeatureNeighborsInputValues -{ - DataPath BoundaryCellsPath; - AttributeMatrixSelectionParameter::ValueType CellFeatureArrayPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - DataPath NeighborListPath; - DataPath NumberOfNeighborsPath; - DataPath SharedSurfaceAreaListPath; - BoolParameter::ValueType StoreBoundaryCells; - BoolParameter::ValueType StoreSurfaceFeatures; - DataPath SurfaceFeaturesPath; -}; - -/** - * @class ComputeFeatureNeighbors - * @brief This algorithm implements support code for the ComputeFeatureNeighborsFilter - */ - -class SIMPLNXCORE_EXPORT ComputeFeatureNeighbors -{ -public: - ComputeFeatureNeighbors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureNeighborsInputValues* inputValues); - ~ComputeFeatureNeighbors() noexcept; - - ComputeFeatureNeighbors(const ComputeFeatureNeighbors&) = delete; - ComputeFeatureNeighbors(ComputeFeatureNeighbors&&) noexcept = delete; - ComputeFeatureNeighbors& operator=(const ComputeFeatureNeighbors&) = delete; - ComputeFeatureNeighbors& operator=(ComputeFeatureNeighbors&&) noexcept = delete; - - Result<> operator()(); - -private: - DataStructure& m_DataStructure; - const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; -}; - -} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp new file mode 100644 index 0000000000..dbe75e0b50 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp @@ -0,0 +1,223 @@ +#include "ComputeFeatureNeighborsDirect.hpp" + +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/NeighborList.hpp" +#include "simplnx/Utilities/MessageHelper.hpp" +#include "simplnx/Utilities/NeighborUtilities.hpp" + +#include + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighborsDirect::ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeFeatureNeighborsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighborsDirect::~ComputeFeatureNeighborsDirect() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> ComputeFeatureNeighborsDirect::operator()() +{ + auto storeBoundaryCells = m_InputValues->StoreBoundaryCells; + auto storeSurfaceFeatures = m_InputValues->StoreSurfaceFeatures; + auto imageGeomPath = m_InputValues->InputImageGeometryPath; + auto featureIdsPath = m_InputValues->FeatureIdsPath; + auto boundaryCellsName = m_InputValues->BoundaryCellsName; + auto numNeighborsName = m_InputValues->NumberOfNeighborsName; + auto neighborListName = m_InputValues->NeighborListName; + auto sharedSurfaceAreaName = m_InputValues->SharedSurfaceAreaListName; + auto surfaceFeaturesName = m_InputValues->SurfaceFeaturesName; + auto featureAttrMatrixPath = m_InputValues->CellFeatureArrayPath; + + DataPath boundaryCellsPath = featureIdsPath.replaceName(boundaryCellsName); + DataPath numNeighborsPath = featureAttrMatrixPath.createChildPath(numNeighborsName); + DataPath neighborListPath = featureAttrMatrixPath.createChildPath(neighborListName); + DataPath sharedSurfaceAreaPath = featureAttrMatrixPath.createChildPath(sharedSurfaceAreaName); + DataPath surfaceFeaturesPath = featureAttrMatrixPath.createChildPath(surfaceFeaturesName); + + auto& featureIds = m_DataStructure.getDataAs(featureIdsPath)->getDataStoreRef(); + auto& numNeighbors = m_DataStructure.getDataAs(numNeighborsPath)->getDataStoreRef(); + + auto& neighborList = m_DataStructure.getDataRefAs(neighborListPath); + auto& sharedSurfaceAreaList = m_DataStructure.getDataRefAs(sharedSurfaceAreaPath); + + auto* boundaryCells = storeBoundaryCells ? m_DataStructure.getDataAs(boundaryCellsPath)->getDataStore() : nullptr; + auto* surfaceFeatures = storeSurfaceFeatures ? m_DataStructure.getDataAs(surfaceFeaturesPath)->getDataStore() : nullptr; + + usize totalPoints = featureIds.getNumberOfTuples(); + usize totalFeatures = numNeighbors.getNumberOfTuples(); + + /* Ensure that we will be able to work with the user selected featureId Array */ + const auto [minFeatureId, maxFeatureId] = std::minmax_element(featureIds.begin(), featureIds.end()); + if(static_cast(*maxFeatureId) >= totalFeatures) + { + std::stringstream out; + out << "Data Array " << featureIdsPath.getTargetName() << " has a maximum value of " << *maxFeatureId << " which is greater than the " + << " number of features from array " << numNeighborsPath.getTargetName() << " which has " << totalFeatures << ". Did you select the " + << " incorrect array for the 'FeatureIds' array?"; + return MakeErrorResult(-24500, out.str()); + } + + auto& imageGeom = m_DataStructure.getDataRefAs(imageGeomPath); + SizeVec3 uDims = imageGeom.getDimensions(); + + std::array dims = { + static_cast(uDims[0]), + static_cast(uDims[1]), + static_cast(uDims[2]), + }; + + std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); + std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); + + int32 feature = 0; + int32 nnum = 0; + uint8 onsurf = 0; + + std::vector> neighborlist(totalFeatures); + std::vector> neighborsurfacearealist(totalFeatures); + + int32 nListSize = 100; + + MessageHelper messageHelper(m_MessageHandler); + ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); + // Initialize the neighbor lists + for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) + { + auto now = std::chrono::steady_clock::now(); + throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Initializing Neighbor Lists || {:.2f}% Complete", CalculatePercentComplete(featureIdx, totalFeatures)); }); + + if(m_ShouldCancel) + { + return {}; + } + + numNeighbors[featureIdx] = 0; + neighborlist[featureIdx].resize(nListSize); + neighborsurfacearealist[featureIdx].assign(nListSize, -1.0f); + if(storeSurfaceFeatures && surfaceFeatures != nullptr) + { + surfaceFeatures->setValue(featureIdx, false); + } + } + + // Loop over all points to generate the neighbor lists + for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) + { + throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Determining Neighbor Lists || {:.2f}% Complete", CalculatePercentComplete(voxelIndex, totalPoints)); }); + + if(m_ShouldCancel) + { + return {}; + } + + onsurf = 0; + feature = featureIds[voxelIndex]; + if(feature > 0 && feature < neighborlist.size()) + { + int64 xIdx = voxelIndex % dims[0]; + int64 yIdx = (voxelIndex / dims[0]) % dims[1]; + int64 zIdx = voxelIndex / (dims[0] * dims[1]); + + if(storeSurfaceFeatures && surfaceFeatures != nullptr) + { + if((xIdx == 0 || xIdx == static_cast((dims[0] - 1)) || yIdx == 0 || yIdx == static_cast((dims[1]) - 1) || zIdx == 0 || zIdx == static_cast((dims[2] - 1))) && dims[2] != 1) + { + surfaceFeatures->setValue(feature, true); + } + if((xIdx == 0 || xIdx == static_cast((dims[0] - 1)) || yIdx == 0 || yIdx == static_cast((dims[1] - 1))) && dims[2] == 1) + { + surfaceFeatures->setValue(feature, true); + } + } + + // Loop over the 6 face neighbors of the voxel + std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + for(const auto& faceIndex : faceNeighborInternalIdx) + { + if(!isValidFaceNeighbor[faceIndex]) + { + continue; + } + + const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + + if(featureIds[neighborPoint] != feature && featureIds[neighborPoint] > 0) + { + onsurf++; + nnum = numNeighbors[feature]; + neighborlist[feature].push_back(featureIds[neighborPoint]); + nnum++; + numNeighbors[feature] = nnum; + } + } + } + if(storeBoundaryCells && boundaryCells != nullptr) + { + boundaryCells->setValue(voxelIndex, static_cast(onsurf)); + } + } + + FloatVec3 spacing = imageGeom.getSpacing(); + + // We do this to create new set of NeighborList objects + for(usize i = 1; i < totalFeatures; i++) + { + throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Calculating Surface Areas || {:.2f}% Complete", CalculatePercentComplete(i, totalFeatures)); }); + + if(m_ShouldCancel) + { + return {}; + } + + std::map neighToCount; + auto numneighs = static_cast(neighborlist[i].size()); + + // this increments the voxel counts for each feature + for(int32 j = 0; j < numneighs; j++) + { + neighToCount[neighborlist[i][j]]++; + } + + auto neighborIter = neighToCount.find(0); + neighToCount.erase(neighborIter); + neighborIter = neighToCount.find(-1); + if(neighborIter != neighToCount.end()) + { + neighToCount.erase(neighborIter); + } + // Resize the features neighbor list to zero + neighborlist[i].resize(0); + neighborsurfacearealist[i].resize(0); + + for(const auto [neigh, number] : neighToCount) + { + float area = static_cast(number) * spacing[0] * spacing[1]; + + // Push the neighbor feature identifier back onto the list, so we stay synced up + neighborlist[i].push_back(neigh); + neighborsurfacearealist[i].push_back(area); + } + numNeighbors[i] = static_cast(neighborlist[i].size()); + + // Set the vector for each list into the NeighborList Object + NeighborList::SharedVectorType sharedNeiLst(new std::vector); + sharedNeiLst->assign(neighborlist[i].begin(), neighborlist[i].end()); + neighborList.setList(static_cast(i), sharedNeiLst); + + NeighborList::SharedVectorType sharedSAL(new std::vector); + sharedSAL->assign(neighborsurfacearealist[i].begin(), neighborsurfacearealist[i].end()); + sharedSurfaceAreaList.setList(static_cast(i), sharedSAL); + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp new file mode 100644 index 0000000000..c4d9d41770 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp @@ -0,0 +1,73 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ArraySelectionParameter.hpp" +#include "simplnx/Parameters/AttributeMatrixSelectionParameter.hpp" +#include "simplnx/Parameters/BoolParameter.hpp" +#include "simplnx/Parameters/DataObjectNameParameter.hpp" +#include "simplnx/Parameters/GeometrySelectionParameter.hpp" + +/** +* This is example code to put in the Execute Method of the filter. + ComputeFeatureNeighborsInputValues inputValues; + inputValues.BoundaryCellsName = filterArgs.value(boundary_cells_name); + inputValues.CellFeatureArrayPath = filterArgs.value(cell_feature_array_path); + inputValues.FeatureIdsPath = filterArgs.value(feature_ids_path); + inputValues.InputImageGeometryPath = filterArgs.value(input_image_geometry_path); + inputValues.NeighborListName = filterArgs.value(neighbor_list_name); + inputValues.NumberOfNeighborsName = filterArgs.value(number_of_neighbors_name); + inputValues.SharedSurfaceAreaListName = filterArgs.value(shared_surface_area_list_name); + inputValues.StoreBoundaryCells = filterArgs.value(store_boundary_cells); + inputValues.StoreSurfaceFeatures = filterArgs.value(store_surface_features); + inputValues.SurfaceFeaturesName = filterArgs.value(surface_features_name); + return ComputeFeatureNeighborsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + +*/ + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT ComputeFeatureNeighborsInputValues +{ + DataObjectNameParameter::ValueType BoundaryCellsName; + AttributeMatrixSelectionParameter::ValueType CellFeatureArrayPath; + ArraySelectionParameter::ValueType FeatureIdsPath; + GeometrySelectionParameter::ValueType InputImageGeometryPath; + DataObjectNameParameter::ValueType NeighborListName; + DataObjectNameParameter::ValueType NumberOfNeighborsName; + DataObjectNameParameter::ValueType SharedSurfaceAreaListName; + BoolParameter::ValueType StoreBoundaryCells; + BoolParameter::ValueType StoreSurfaceFeatures; + DataObjectNameParameter::ValueType SurfaceFeaturesName; +}; + +/** + * @class ComputeFeatureNeighborsDirect + * @brief This algorithm implements support code for the ComputeFeatureNeighborsFilter + */ + +class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsDirect +{ +public: + ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureNeighborsInputValues* inputValues); + ~ComputeFeatureNeighborsDirect() noexcept; + + ComputeFeatureNeighborsDirect(const ComputeFeatureNeighborsDirect&) = delete; + ComputeFeatureNeighborsDirect(ComputeFeatureNeighborsDirect&&) noexcept = delete; + ComputeFeatureNeighborsDirect& operator=(const ComputeFeatureNeighborsDirect&) = delete; + ComputeFeatureNeighborsDirect& operator=(ComputeFeatureNeighborsDirect&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp similarity index 90% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp index 62c870f18b..4676e42c31 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp @@ -1,4 +1,4 @@ -#include "ComputeKMedoids.hpp" +#include "ComputeKMedoidsDirect.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/Utilities/ClusteringUtilities.hpp" @@ -15,7 +15,7 @@ template class KMedoidsTemplate { public: - KMedoidsTemplate(ComputeKMedoids* filter, const IDataArray* inputIDataArray, IDataArray* medoidsIDataArray, const std::unique_ptr& maskDataArray, + KMedoidsTemplate(ComputeKMedoidsDirect* filter, const IDataArray* inputIDataArray, IDataArray* medoidsIDataArray, const std::unique_ptr& maskDataArray, usize numClusters, Int32AbstractDataStore& fIds, ClusterUtilities::DistanceMetric distMetric, std::mt19937_64::result_type seed) : m_Filter(filter) , m_InputArray(inputIDataArray->template getIDataStoreRefAs>()) @@ -90,7 +90,7 @@ class KMedoidsTemplate private: using DataArrayT = DataArray; using AbstractDataStoreT = AbstractDataStore; - ComputeKMedoids* m_Filter; + ComputeKMedoidsDirect* m_Filter; const AbstractDataStoreT& m_InputArray; AbstractDataStoreT& m_Medoids; const std::unique_ptr& m_Mask; @@ -182,7 +182,7 @@ class KMedoidsTemplate } // namespace // ----------------------------------------------------------------------------- -ComputeKMedoids::ComputeKMedoids(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues) +ComputeKMedoidsDirect::ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -191,22 +191,22 @@ ComputeKMedoids::ComputeKMedoids(DataStructure& dataStructure, const IFilter::Me } // ----------------------------------------------------------------------------- -ComputeKMedoids::~ComputeKMedoids() noexcept = default; +ComputeKMedoidsDirect::~ComputeKMedoidsDirect() noexcept = default; // ----------------------------------------------------------------------------- -void ComputeKMedoids::updateProgress(const std::string& message) +void ComputeKMedoidsDirect::updateProgress(const std::string& message) { m_MessageHandler(IFilter::Message::Type::Info, message); } // ----------------------------------------------------------------------------- -const std::atomic_bool& ComputeKMedoids::getCancel() +const std::atomic_bool& ComputeKMedoidsDirect::getCancel() { return m_ShouldCancel; } // ----------------------------------------------------------------------------- -Result<> ComputeKMedoids::operator()() +Result<> ComputeKMedoidsDirect::operator()() { auto* clusteringArray = m_DataStructure.getDataAs(m_InputValues->ClusteringArrayPath); std::unique_ptr maskCompare; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp similarity index 63% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp index 745b660209..8f51562a63 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp @@ -24,16 +24,16 @@ struct SIMPLNXCORE_EXPORT KMedoidsInputValues /** * @class */ -class SIMPLNXCORE_EXPORT ComputeKMedoids +class SIMPLNXCORE_EXPORT ComputeKMedoidsDirect { public: - ComputeKMedoids(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues); - ~ComputeKMedoids() noexcept; + ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues); + ~ComputeKMedoidsDirect() noexcept; - ComputeKMedoids(const ComputeKMedoids&) = delete; - ComputeKMedoids(ComputeKMedoids&&) noexcept = delete; - ComputeKMedoids& operator=(const ComputeKMedoids&) = delete; - ComputeKMedoids& operator=(ComputeKMedoids&&) noexcept = delete; + ComputeKMedoidsDirect(const ComputeKMedoidsDirect&) = delete; + ComputeKMedoidsDirect(ComputeKMedoidsDirect&&) noexcept = delete; + ComputeKMedoidsDirect& operator=(const ComputeKMedoidsDirect&) = delete; + ComputeKMedoidsDirect& operator=(ComputeKMedoidsDirect&&) noexcept = delete; Result<> operator()(); void updateProgress(const std::string& message); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp similarity index 93% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp index 36518d9b30..f08c4de596 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp @@ -1,4 +1,4 @@ -#include "ComputeSurfaceAreaToVolume.hpp" +#include "ComputeSurfaceAreaToVolumeDirect.hpp" #include "simplnx/Common/Constants.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -9,7 +9,7 @@ using namespace nx::core; // ----------------------------------------------------------------------------- -ComputeSurfaceAreaToVolume::ComputeSurfaceAreaToVolume(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, +ComputeSurfaceAreaToVolumeDirect::ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) @@ -19,16 +19,16 @@ ComputeSurfaceAreaToVolume::ComputeSurfaceAreaToVolume(DataStructure& dataStruct } // ----------------------------------------------------------------------------- -ComputeSurfaceAreaToVolume::~ComputeSurfaceAreaToVolume() noexcept = default; +ComputeSurfaceAreaToVolumeDirect::~ComputeSurfaceAreaToVolumeDirect() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ComputeSurfaceAreaToVolume::getCancel() +const std::atomic_bool& ComputeSurfaceAreaToVolumeDirect::getCancel() { return m_ShouldCancel; } // ----------------------------------------------------------------------------- -Result<> ComputeSurfaceAreaToVolume::operator()() +Result<> ComputeSurfaceAreaToVolumeDirect::operator()() { // Input Cell Data auto featureIdsArrayPtr = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp similarity index 54% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp index 637caf9980..560ef2c1c6 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp @@ -22,16 +22,16 @@ struct SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeInputValues /** * @class */ -class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolume +class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeDirect { public: - ComputeSurfaceAreaToVolume(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues); - ~ComputeSurfaceAreaToVolume() noexcept; + ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues); + ~ComputeSurfaceAreaToVolumeDirect() noexcept; - ComputeSurfaceAreaToVolume(const ComputeSurfaceAreaToVolume&) = delete; - ComputeSurfaceAreaToVolume(ComputeSurfaceAreaToVolume&&) noexcept = delete; - ComputeSurfaceAreaToVolume& operator=(const ComputeSurfaceAreaToVolume&) = delete; - ComputeSurfaceAreaToVolume& operator=(ComputeSurfaceAreaToVolume&&) noexcept = delete; + ComputeSurfaceAreaToVolumeDirect(const ComputeSurfaceAreaToVolumeDirect&) = delete; + ComputeSurfaceAreaToVolumeDirect(ComputeSurfaceAreaToVolumeDirect&&) noexcept = delete; + ComputeSurfaceAreaToVolumeDirect& operator=(const ComputeSurfaceAreaToVolumeDirect&) = delete; + ComputeSurfaceAreaToVolumeDirect& operator=(ComputeSurfaceAreaToVolumeDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp similarity index 95% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp index 9dcf86a149..82b9006d49 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp @@ -1,4 +1,4 @@ -#include "ComputeSurfaceFeatures.hpp" +#include "ComputeSurfaceFeaturesDirect.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -181,7 +181,7 @@ void findSurfaceFeatures2D(DataStructure& dataStructure, const DataPath& feature } // namespace // ----------------------------------------------------------------------------- -ComputeSurfaceFeatures::ComputeSurfaceFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, +ComputeSurfaceFeaturesDirect::ComputeSurfaceFeaturesDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) @@ -191,10 +191,10 @@ ComputeSurfaceFeatures::ComputeSurfaceFeatures(DataStructure& dataStructure, con } // ----------------------------------------------------------------------------- -ComputeSurfaceFeatures::~ComputeSurfaceFeatures() noexcept = default; +ComputeSurfaceFeaturesDirect::~ComputeSurfaceFeaturesDirect() noexcept = default; // ----------------------------------------------------------------------------- -Result<> ComputeSurfaceFeatures::operator()() +Result<> ComputeSurfaceFeaturesDirect::operator()() { const auto pMarkFeature0NeighborsValue = m_InputValues->MarkFeature0Neighbors; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp similarity index 65% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp index 0c825a2b1b..8be8b9087c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp @@ -24,20 +24,20 @@ struct SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesInputValues }; /** - * @class ComputeSurfaceFeatures + * @class ComputeSurfaceFeaturesDirect * @brief This algorithm implements support code for the ComputeSurfaceFeaturesFilter */ -class SIMPLNXCORE_EXPORT ComputeSurfaceFeatures +class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesDirect { public: - ComputeSurfaceFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues); - ~ComputeSurfaceFeatures() noexcept; + ComputeSurfaceFeaturesDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues); + ~ComputeSurfaceFeaturesDirect() noexcept; - ComputeSurfaceFeatures(const ComputeSurfaceFeatures&) = delete; - ComputeSurfaceFeatures(ComputeSurfaceFeatures&&) noexcept = delete; - ComputeSurfaceFeatures& operator=(const ComputeSurfaceFeatures&) = delete; - ComputeSurfaceFeatures& operator=(ComputeSurfaceFeatures&&) noexcept = delete; + ComputeSurfaceFeaturesDirect(const ComputeSurfaceFeaturesDirect&) = delete; + ComputeSurfaceFeaturesDirect(ComputeSurfaceFeaturesDirect&&) noexcept = delete; + ComputeSurfaceFeaturesDirect& operator=(const ComputeSurfaceFeaturesDirect&) = delete; + ComputeSurfaceFeaturesDirect& operator=(ComputeSurfaceFeaturesDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp similarity index 97% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp index 342cab6501..35547bcbe5 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp @@ -1,4 +1,4 @@ -#include "DBSCAN.hpp" +#include "DBSCANDirect.hpp" #include "simplnx/Common/Range.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" @@ -18,7 +18,7 @@ namespace * Implementation derived from: https://yliu.site/pub/GDCF_PR2019.pdf * Citation: * Thapana Boonchoo, Xiang Ao, Yang Liu, Weizhong Zhao, Fuzhen Zhuang, Qing He, - * Grid-based DBSCAN: Indexing and inference, + * Grid-based DBSCANDirect: Indexing and inference, * https://doi.org/10.1016/j.patcog.2019.01.034. * * Definitions: @@ -716,7 +716,7 @@ class GDCF { } - Result<> cluster(usize minPoints, DBSCAN::ParseOrder parseOrder, std::mt19937_64::result_type seed = std::mt19937_64::default_seed) + Result<> cluster(usize minPoints, DBSCANDirect::ParseOrder parseOrder, std::mt19937_64::result_type seed = std::mt19937_64::default_seed) { m_MessageHelper.sendMessage(" - Identifying core grids..."); // Identify Core Grids @@ -742,11 +742,11 @@ class GDCF m_MessageHelper.sendMessage(" - Sorting grids according to supplied parse order..."); switch(parseOrder) { - case DBSCAN::ParseOrder::LowDensityFirst: { + case DBSCANDirect::ParseOrder::LowDensityFirst: { QuickSortGrids(coreGridIds, 0, coreGridIds.size() - 1); break; } - case DBSCAN::ParseOrder::Random: { + case DBSCANDirect::ParseOrder::Random: { std::mt19937_64 gen(seed); std::uniform_real_distribution dist(0, 1); @@ -762,7 +762,7 @@ class GDCF break; } - case DBSCAN::SeededRandom: { + case DBSCANDirect::SeededRandom: { std::mt19937_64 gen(seed); std::uniform_real_distribution dist(0, 1); @@ -1046,7 +1046,7 @@ Result<> RunAlgorithm(const DBSCANInputValues* inputValues, const AbstractDataSt } messageHelper.sendMessage("Clustering:"); - Result<> result = algorithm.cluster(inputValues->MinPoints, static_cast(inputValues->ParseOrder), inputValues->Seed); + Result<> result = algorithm.cluster(inputValues->MinPoints, static_cast(inputValues->ParseOrder), inputValues->Seed); if(result.invalid() || !result.warnings().empty()) { // If the result has warnings in it the cluster forest is @@ -1063,7 +1063,7 @@ Result<> RunAlgorithm(const DBSCANInputValues* inputValues, const AbstractDataSt return algorithm.label(featureIds.getDataStoreRef()); } -struct DBSCANFunctor +struct DBSCANDirectFunctor { template Result<> operator()(const DBSCANInputValues* inputValues, const IDataArray& clusterArray, const std::unique_ptr& mask, Int32Array& featureIds, @@ -1089,7 +1089,7 @@ struct DBSCANFunctor } // namespace // ----------------------------------------------------------------------------- -DBSCAN::DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues) +DBSCANDirect::DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -1098,10 +1098,10 @@ DBSCAN::DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesg } // ----------------------------------------------------------------------------- -DBSCAN::~DBSCAN() noexcept = default; +DBSCANDirect::~DBSCANDirect() noexcept = default; // ----------------------------------------------------------------------------- -Result<> DBSCAN::operator()() +Result<> DBSCANDirect::operator()() { MessageHelper messageHelper(m_MessageHandler); @@ -1120,7 +1120,7 @@ Result<> DBSCAN::operator()() return MakeErrorResult(-54060, message); } - Result<> result = ExecuteDataFunction(DBSCANFunctor{}, clusteringArray.getDataType(), m_InputValues, clusteringArray, maskCompare, featureIds, messageHelper, m_ShouldCancel); + Result<> result = ExecuteDataFunction(DBSCANDirectFunctor{}, clusteringArray.getDataType(), m_InputValues, clusteringArray, maskCompare, featureIds, messageHelper, m_ShouldCancel); if(result.invalid()) { return result; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp similarity index 69% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp index f2e6d9e070..16f836ac4c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp @@ -28,16 +28,16 @@ struct SIMPLNXCORE_EXPORT DBSCANInputValues /** * @class */ -class SIMPLNXCORE_EXPORT DBSCAN +class SIMPLNXCORE_EXPORT DBSCANDirect { public: - DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues); - ~DBSCAN() noexcept; + DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues); + ~DBSCANDirect() noexcept; - DBSCAN(const DBSCAN&) = delete; - DBSCAN(DBSCAN&&) noexcept = delete; - DBSCAN& operator=(const DBSCAN&) = delete; - DBSCAN& operator=(DBSCAN&&) noexcept = delete; + DBSCANDirect(const DBSCANDirect&) = delete; + DBSCANDirect(DBSCANDirect&&) noexcept = delete; + DBSCANDirect& operator=(const DBSCANDirect&) = delete; + DBSCANDirect& operator=(DBSCANDirect&&) noexcept = delete; enum ParseOrder { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp similarity index 94% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp index 7cf31a305a..6eb72ef971 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp @@ -1,4 +1,4 @@ -#include "FillBadData.hpp" +#include "FillBadDataBFS.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -13,7 +13,7 @@ using namespace nx::core; // ============================================================================= -// FillBadData Algorithm Overview +// FillBadDataBFS Algorithm Overview // ============================================================================= // // This file implements an optimized algorithm for filling bad data (voxels with @@ -52,7 +52,7 @@ namespace // @param outputDataStore The data array to update // @param neighbors The neighbor assignments (index of the neighbor to copy from) template -void FillBadDataUpdateTuples(const Int32AbstractDataStore& featureIds, AbstractDataStore& outputDataStore, const std::vector& neighbors) +void FillBadDataBFSUpdateTuples(const Int32AbstractDataStore& featureIds, AbstractDataStore& outputDataStore, const std::vector& neighbors) { usize start = 0; usize stop = outputDataStore.getNumberOfTuples(); @@ -87,14 +87,14 @@ void FillBadDataUpdateTuples(const Int32AbstractDataStore& featureIds, AbstractD // ----------------------------------------------------------------------------- // Functor for type-dispatched tuple updates // ----------------------------------------------------------------------------- -// Allows the FillBadDataUpdateTuples function to be called with runtime type dispatch -struct FillBadDataUpdateTuplesFunctor +// Allows the FillBadDataBFSUpdateTuples function to be called with runtime type dispatch +struct FillBadDataBFSUpdateTuplesFunctor { template void operator()(const Int32AbstractDataStore& featureIds, IDataArray* outputIDataArray, const std::vector& neighbors) { auto& outputStore = outputIDataArray->template getIDataStoreRefAs>(); - FillBadDataUpdateTuples(featureIds, outputStore, neighbors); + FillBadDataBFSUpdateTuples(featureIds, outputStore, neighbors); } }; } // namespace @@ -256,10 +256,10 @@ void ChunkAwareUnionFind::flatten() } // ============================================================================= -// FillBadData Implementation +// FillBadDataBFS Implementation // ============================================================================= -FillBadData::FillBadData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, FillBadDataInputValues* inputValues) +FillBadDataBFS::FillBadDataBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, FillBadDataInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -268,10 +268,10 @@ FillBadData::FillBadData(DataStructure& dataStructure, const IFilter::MessageHan } // ----------------------------------------------------------------------------- -FillBadData::~FillBadData() noexcept = default; +FillBadDataBFS::~FillBadDataBFS() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& FillBadData::getCancel() const +const std::atomic_bool& FillBadDataBFS::getCancel() const { return m_ShouldCancel; } @@ -300,7 +300,7 @@ const std::atomic_bool& FillBadData::getCancel() const // @param provisionalLabels Map from voxel index to assigned provisional label // @param dims Image dimensions [X, Y, Z] // ============================================================================= -Result<> FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims) +Result<> FillBadDataBFS::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims) { int64 nextLabel = -1; const usize slabSize = static_cast(dims[0]) * static_cast(dims[1]); @@ -402,7 +402,7 @@ Result<> FillBadData::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, Chunk // @param unionFind Union-Find structure containing label equivalences // @param smallRegions Unused in current implementation (kept for interface compatibility) // ============================================================================= -void FillBadData::phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std::unordered_set& smallRegions) +void FillBadDataBFS::phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std::unordered_set& smallRegions) { // Flatten the union-find structure to: // 1. Compress all paths (make every label point directly to root) @@ -428,8 +428,8 @@ void FillBadData::phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std:: // @param unionFind Union-Find structure with resolved equivalences (from Phase 2) // @param maxPhase Maximum existing phase value (for new phase assignment) // ============================================================================= -Result<> FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, - const std::unordered_set& /*smallRegions*/, ChunkAwareUnionFind& unionFind, usize maxPhase) const +Result<> FillBadDataBFS::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, + const std::unordered_set& /*smallRegions*/, ChunkAwareUnionFind& unionFind, usize maxPhase) const { // Classify regions by size std::unordered_map rootSizes; @@ -524,7 +524,7 @@ Result<> FillBadData::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsSto // @param dims Image dimensions [X, Y, Z] // @param numFeatures Number of features in the dataset // ============================================================================= -void FillBadData::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, usize numFeatures) const +void FillBadDataBFS::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, usize numFeatures) const { const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); const usize totalPoints = featureIdsStore.getNumberOfTuples(); @@ -642,11 +642,11 @@ void FillBadData::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore auto* oldCellArray = m_DataStructure.getDataAs(cellArrayPath); // Use the type-dispatched update function to handle all data types - ExecuteDataFunction(FillBadDataUpdateTuplesFunctor{}, oldCellArray->getDataType(), featureIdsStore, oldCellArray, neighbors); + ExecuteDataFunction(FillBadDataBFSUpdateTuplesFunctor{}, oldCellArray->getDataType(), featureIdsStore, oldCellArray, neighbors); } // Update FeatureIds array last to finalize the iteration - FillBadDataUpdateTuples(featureIdsStore, featureIdsStore, neighbors); + FillBadDataBFSUpdateTuples(featureIdsStore, featureIdsStore, neighbors); // Send throttled progress update (max 1 per second) throttledMessenger.sendThrottledMessage([iteration, count]() { return fmt::format(" Iteration {}: {} voxels remaining to fill", iteration, count); }); @@ -668,7 +668,7 @@ void FillBadData::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore // // @return Result indicating success or failure // ============================================================================= -Result<> FillBadData::operator()() const +Result<> FillBadDataBFS::operator()() const { auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath)->getDataStoreRef(); const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp similarity index 89% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp index e40989040e..e7679dd366 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp @@ -83,19 +83,19 @@ struct SIMPLNXCORE_EXPORT FillBadDataInputValues }; /** - * @class FillBadData + * @class FillBadDataBFS */ -class SIMPLNXCORE_EXPORT FillBadData +class SIMPLNXCORE_EXPORT FillBadDataBFS { public: - FillBadData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, FillBadDataInputValues* inputValues); - ~FillBadData() noexcept; + FillBadDataBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, FillBadDataInputValues* inputValues); + ~FillBadDataBFS() noexcept; - FillBadData(const FillBadData&) = delete; - FillBadData(FillBadData&&) noexcept = delete; - FillBadData& operator=(const FillBadData&) = delete; - FillBadData& operator=(FillBadData&&) noexcept = delete; + FillBadDataBFS(const FillBadDataBFS&) = delete; + FillBadDataBFS(FillBadDataBFS&&) noexcept = delete; + FillBadDataBFS& operator=(const FillBadDataBFS&) = delete; + FillBadDataBFS& operator=(FillBadDataBFS&&) noexcept = delete; Result<> operator()() const; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp similarity index 93% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp index 00ee05cc31..954016d883 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp @@ -1,4 +1,4 @@ -#include "IdentifySample.hpp" +#include "IdentifySampleBFS.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -9,7 +9,7 @@ using namespace nx::core; namespace { -struct IdentifySampleFunctor +struct IdentifySampleBFSFunctor { template void operator()(const ImageGeom* imageGeom, IDataArray* goodVoxelsPtr, bool fillHoles, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel) @@ -177,7 +177,7 @@ struct IdentifySampleFunctor } }; -struct IdentifySampleSliceBySliceFunctor +struct IdentifySampleBFSSliceBySliceFunctor { enum class Plane { @@ -390,7 +390,7 @@ struct IdentifySampleSliceBySliceFunctor } // namespace // ----------------------------------------------------------------------------- -IdentifySample::IdentifySample(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues) +IdentifySampleBFS::IdentifySampleBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -399,22 +399,22 @@ IdentifySample::IdentifySample(DataStructure& dataStructure, const IFilter::Mess } // ----------------------------------------------------------------------------- -IdentifySample::~IdentifySample() noexcept = default; +IdentifySampleBFS::~IdentifySampleBFS() noexcept = default; // ----------------------------------------------------------------------------- -Result<> IdentifySample::operator()() +Result<> IdentifySampleBFS::operator()() { auto* inputData = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); const auto* imageGeom = m_DataStructure.getDataAs(m_InputValues->InputImageGeometryPath); if(m_InputValues->SliceBySlice) { - ExecuteDataFunction(IdentifySampleSliceBySliceFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, - static_cast(m_InputValues->SliceBySlicePlaneIndex), m_MessageHandler, m_ShouldCancel); + ExecuteDataFunction(IdentifySampleBFSSliceBySliceFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, + static_cast(m_InputValues->SliceBySlicePlaneIndex), m_MessageHandler, m_ShouldCancel); } else { - ExecuteDataFunction(IdentifySampleFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, m_MessageHandler, m_ShouldCancel); + ExecuteDataFunction(IdentifySampleBFSFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, m_MessageHandler, m_ShouldCancel); } return {}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp similarity index 67% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp index a24a219f25..57a8f93553 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp @@ -23,20 +23,20 @@ struct SIMPLNXCORE_EXPORT IdentifySampleInputValues }; /** - * @class IdentifySample + * @class IdentifySampleBFS * @brief This algorithm implements support code for the IdentifySampleFilter */ -class SIMPLNXCORE_EXPORT IdentifySample +class SIMPLNXCORE_EXPORT IdentifySampleBFS { public: - IdentifySample(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues); - ~IdentifySample() noexcept; + IdentifySampleBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues); + ~IdentifySampleBFS() noexcept; - IdentifySample(const IdentifySample&) = delete; - IdentifySample(IdentifySample&&) noexcept = delete; - IdentifySample& operator=(const IdentifySample&) = delete; - IdentifySample& operator=(IdentifySample&&) noexcept = delete; + IdentifySampleBFS(const IdentifySampleBFS&) = delete; + IdentifySampleBFS(IdentifySampleBFS&&) noexcept = delete; + IdentifySampleBFS& operator=(const IdentifySampleBFS&) = delete; + IdentifySampleBFS& operator=(IdentifySampleBFS&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp similarity index 96% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp index 48ea54b9d7..fcf95d0b0d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp @@ -1,4 +1,4 @@ -#include "MultiThresholdObjects.hpp" +#include "MultiThresholdObjectsDirect.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -220,7 +220,7 @@ struct ThresholdSetFunctor } // namespace // ----------------------------------------------------------------------------- -MultiThresholdObjects::MultiThresholdObjects(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, +MultiThresholdObjectsDirect::MultiThresholdObjectsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) @@ -230,10 +230,10 @@ MultiThresholdObjects::MultiThresholdObjects(DataStructure& dataStructure, const } // ----------------------------------------------------------------------------- -MultiThresholdObjects::~MultiThresholdObjects() noexcept = default; +MultiThresholdObjectsDirect::~MultiThresholdObjectsDirect() noexcept = default; // ----------------------------------------------------------------------------- -Result<> MultiThresholdObjects::operator()() +Result<> MultiThresholdObjectsDirect::operator()() { auto thresholdsObject = m_InputValues->ArrayThresholdsObject; auto maskArrayName = m_InputValues->OutputDataArrayName; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp similarity index 73% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp index 2371efc102..eaef405c9f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp @@ -21,7 +21,7 @@ inputValues.OutputDataArrayName = filterArgs.value(output_data_array_name); inputValues.UseCustomFalseValue = filterArgs.value(use_custom_false_value); inputValues.UseCustomTrueValue = filterArgs.value(use_custom_true_value); - return MultiThresholdObjects(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return MultiThresholdObjectsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); */ @@ -40,20 +40,20 @@ struct SIMPLNXCORE_EXPORT MultiThresholdObjectsInputValues }; /** - * @class MultiThresholdObjects + * @class MultiThresholdObjectsDirect * @brief This algorithm implements support code for the MultiThresholdObjectsFilter */ -class SIMPLNXCORE_EXPORT MultiThresholdObjects +class SIMPLNXCORE_EXPORT MultiThresholdObjectsDirect { public: - MultiThresholdObjects(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues); - ~MultiThresholdObjects() noexcept; + MultiThresholdObjectsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues); + ~MultiThresholdObjectsDirect() noexcept; - MultiThresholdObjects(const MultiThresholdObjects&) = delete; - MultiThresholdObjects(MultiThresholdObjects&&) noexcept = delete; - MultiThresholdObjects& operator=(const MultiThresholdObjects&) = delete; - MultiThresholdObjects& operator=(MultiThresholdObjects&&) noexcept = delete; + MultiThresholdObjectsDirect(const MultiThresholdObjectsDirect&) = delete; + MultiThresholdObjectsDirect(MultiThresholdObjectsDirect&&) noexcept = delete; + MultiThresholdObjectsDirect& operator=(const MultiThresholdObjectsDirect&) = delete; + MultiThresholdObjectsDirect& operator=(MultiThresholdObjectsDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp similarity index 97% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp index 10ffc4c42f..3efec65f45 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp @@ -1,4 +1,4 @@ -#include "QuickSurfaceMesh.hpp" +#include "QuickSurfaceMeshDirect.hpp" #include "TupleTransfer.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -65,7 +65,7 @@ using EdgeMap = std::unordered_mapgetOrigin()) @@ -225,17 +225,17 @@ struct GenerateTripleLinesImpl }; // ----------------------------------------------------------------------------- -void GetGridCoordinates(const IGridGeometry* grid, size_t x, size_t y, size_t z, QuickSurfaceMesh::VertexStore& verts, IGeometry::MeshIndexType nodeIndex) +void GetGridCoordinates(const IGridGeometry* grid, size_t x, size_t y, size_t z, QuickSurfaceMeshDirect::VertexStore& verts, IGeometry::MeshIndexType nodeIndex) { nx::core::Point3D tmpCoords = grid->getPlaneCoords(x, y, z); - verts[nodeIndex] = static_cast(tmpCoords[0]); - verts[nodeIndex + 1] = static_cast(tmpCoords[1]); - verts[nodeIndex + 2] = static_cast(tmpCoords[2]); + verts[nodeIndex] = static_cast(tmpCoords[0]); + verts[nodeIndex + 1] = static_cast(tmpCoords[1]); + verts[nodeIndex + 2] = static_cast(tmpCoords[2]); } // ----------------------------------------------------------------------------- -void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMesh::MeshIndexType v1, QuickSurfaceMesh::MeshIndexType v2, QuickSurfaceMesh::MeshIndexType v3, - QuickSurfaceMesh::MeshIndexType v4, QuickSurfaceMesh::MeshIndexType v5, QuickSurfaceMesh::MeshIndexType v6) +void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3, + QuickSurfaceMeshDirect::MeshIndexType v4, QuickSurfaceMeshDirect::MeshIndexType v5, QuickSurfaceMeshDirect::MeshIndexType v6) { auto val = static_cast(k_Distribution(k_Generator)); // Random remaining position. @@ -258,8 +258,8 @@ void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMesh: } // ----------------------------------------------------------------------------- -void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMesh::MeshIndexType v1, QuickSurfaceMesh::MeshIndexType v2, QuickSurfaceMesh::MeshIndexType v3, - QuickSurfaceMesh::MeshIndexType v4) +void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3, + QuickSurfaceMeshDirect::MeshIndexType v4) { auto val = static_cast(k_Distribution(k_Generator)); // Random remaining position. @@ -298,7 +298,7 @@ void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMesh: } // ----------------------------------------------------------------------------- -void FlipProblemVoxelCase3(Int32AbstractDataStore& featureIds, QuickSurfaceMesh::MeshIndexType v1, QuickSurfaceMesh::MeshIndexType v2, QuickSurfaceMesh::MeshIndexType v3) +void FlipProblemVoxelCase3(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3) { auto val = static_cast(k_Distribution(k_Generator)); // Random remaining position. @@ -314,7 +314,7 @@ void FlipProblemVoxelCase3(Int32AbstractDataStore& featureIds, QuickSurfaceMesh: } // namespace // ----------------------------------------------------------------------------- -QuickSurfaceMesh::QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler) +QuickSurfaceMeshDirect::QuickSurfaceMeshDirect(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -324,10 +324,10 @@ QuickSurfaceMesh::QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMes } // ----------------------------------------------------------------------------- -QuickSurfaceMesh::~QuickSurfaceMesh() noexcept = default; +QuickSurfaceMeshDirect::~QuickSurfaceMeshDirect() noexcept = default; // ----------------------------------------------------------------------------- -Result<> QuickSurfaceMesh::operator()() +Result<> QuickSurfaceMeshDirect::operator()() { // Get the ImageGeometry auto& grid = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); @@ -480,7 +480,7 @@ Result<> QuickSurfaceMesh::operator()() } // ----------------------------------------------------------------------------- -void QuickSurfaceMesh::correctProblemVoxels() +void QuickSurfaceMeshDirect::correctProblemVoxels() { m_MessageHandler(IFilter::Message::Type::Info, "Correcting Problem Voxels"); @@ -651,7 +651,7 @@ void QuickSurfaceMesh::correctProblemVoxels() } // ----------------------------------------------------------------------------- -void QuickSurfaceMesh::determineActiveNodes(std::vector& nodeIds, MeshIndexType& nodeCount, MeshIndexType& triangleCount) +void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& nodeIds, MeshIndexType& nodeCount, MeshIndexType& triangleCount) { m_MessageHandler(IFilter::Message::Type::Info, "Determining active Nodes"); @@ -954,7 +954,7 @@ void QuickSurfaceMesh::determineActiveNodes(std::vector& nodeIds, } // ----------------------------------------------------------------------------- -void QuickSurfaceMesh::createNodesAndTriangles(std::vector& m_NodeIds, MeshIndexType nodeCount, MeshIndexType triangleCount) +void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& m_NodeIds, MeshIndexType nodeCount, MeshIndexType triangleCount) { if(m_ShouldCancel) { @@ -1009,8 +1009,8 @@ void QuickSurfaceMesh::createNodesAndTriangles(std::vector& m_Nod auto& nodeTypes = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); nodeTypes.resizeTuples({nodeCount}); - QuickSurfaceMesh::VertexStore& vertex = triangleGeom->getVertices()->getDataStoreRef(); - QuickSurfaceMesh::TriStore& triangle = triangleGeom->getFaces()->getDataStoreRef(); + QuickSurfaceMeshDirect::VertexStore& vertex = triangleGeom->getVertices()->getDataStoreRef(); + QuickSurfaceMeshDirect::TriStore& triangle = triangleGeom->getFaces()->getDataStoreRef(); ownerLists.resize(nodeCount); @@ -1554,7 +1554,7 @@ void QuickSurfaceMesh::createNodesAndTriangles(std::vector& m_Nod } // ----------------------------------------------------------------------------- -void QuickSurfaceMesh::generateTripleLines() +void QuickSurfaceMeshDirect::generateTripleLines() { if(m_ShouldCancel) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp similarity index 78% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp index 27bf7da0fa..b41e37a472 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp @@ -32,20 +32,20 @@ struct SIMPLNXCORE_EXPORT QuickSurfaceMeshInputValues MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; }; -class SIMPLNXCORE_EXPORT QuickSurfaceMesh +class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect { public: using VertexStore = AbstractDataStore; using TriStore = AbstractDataStore; using MeshIndexType = IGeometry::MeshIndexType; - QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler); - ~QuickSurfaceMesh() noexcept; + QuickSurfaceMeshDirect(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler); + ~QuickSurfaceMeshDirect() noexcept; - QuickSurfaceMesh(const QuickSurfaceMesh&) = delete; - QuickSurfaceMesh(QuickSurfaceMesh&&) noexcept = delete; - QuickSurfaceMesh& operator=(const QuickSurfaceMesh&) = delete; - QuickSurfaceMesh& operator=(QuickSurfaceMesh&&) noexcept = delete; + QuickSurfaceMeshDirect(const QuickSurfaceMeshDirect&) = delete; + QuickSurfaceMeshDirect(QuickSurfaceMeshDirect&&) noexcept = delete; + QuickSurfaceMeshDirect& operator=(const QuickSurfaceMeshDirect&) = delete; + QuickSurfaceMeshDirect& operator=(QuickSurfaceMeshDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp similarity index 97% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp index ac95808ad2..289d5ed44b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp @@ -1,4 +1,4 @@ -#include "SurfaceNets.hpp" +#include "SurfaceNetsDirect.hpp" #include "TupleTransfer.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -88,7 +88,7 @@ void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing } } // namespace // ----------------------------------------------------------------------------- -SurfaceNets::SurfaceNets(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues) +SurfaceNetsDirect::SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -97,16 +97,16 @@ SurfaceNets::SurfaceNets(DataStructure& dataStructure, const IFilter::MessageHan } // ----------------------------------------------------------------------------- -SurfaceNets::~SurfaceNets() noexcept = default; +SurfaceNetsDirect::~SurfaceNetsDirect() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& SurfaceNets::getCancel() +const std::atomic_bool& SurfaceNetsDirect::getCancel() { return m_ShouldCancel; } // ----------------------------------------------------------------------------- -Result<> SurfaceNets::operator()() +Result<> SurfaceNetsDirect::operator()() { // Get the ImageGeometry auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp similarity index 70% rename from src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp rename to src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp index 27a7c621ef..50aa0f8465 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp @@ -33,16 +33,16 @@ struct SIMPLNXCORE_EXPORT SurfaceNetsInputValues /** * @class */ -class SIMPLNXCORE_EXPORT SurfaceNets +class SIMPLNXCORE_EXPORT SurfaceNetsDirect { public: - SurfaceNets(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues); - ~SurfaceNets() noexcept; + SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues); + ~SurfaceNetsDirect() noexcept; - SurfaceNets(const SurfaceNets&) = delete; - SurfaceNets(SurfaceNets&&) noexcept = delete; - SurfaceNets& operator=(const SurfaceNets&) = delete; - SurfaceNets& operator=(SurfaceNets&&) noexcept = delete; + SurfaceNetsDirect(const SurfaceNetsDirect&) = delete; + SurfaceNetsDirect(SurfaceNetsDirect&&) noexcept = delete; + SurfaceNetsDirect& operator=(const SurfaceNetsDirect&) = delete; + SurfaceNetsDirect& operator=(SurfaceNetsDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp index e18a2a6888..9827ca8563 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeBoundaryCellsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataPath.hpp" @@ -126,7 +126,7 @@ Result<> ComputeBoundaryCellsFilter::executeImpl(DataStructure& dataStructure, c inputValues.FeatureIdsArrayPath = filterArgs.value(k_FeatureIdsArrayPath_Key); inputValues.BoundaryCellsArrayName = inputValues.FeatureIdsArrayPath.replaceName(filterArgs.value(k_BoundaryCellsArrayName_Key)); - return ComputeBoundaryCells(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeBoundaryCellsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp index b518d3d59d..ceaaaae48c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeFeatureNeighborsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -189,7 +189,7 @@ Result<> ComputeFeatureNeighborsFilter::executeImpl(DataStructure& dataStructure inputValues.SharedSurfaceAreaListPath = inputValues.CellFeatureArrayPath.createChildPath(filterArgs.value(k_SharedSurfaceAreaName_Key)); inputValues.SurfaceFeaturesPath = inputValues.CellFeatureArrayPath.createChildPath(filterArgs.value(k_SurfaceFeaturesName_Key)); - return ComputeFeatureNeighbors(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeFeatureNeighborsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp index 556944f6c8..70e3287dd0 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeKMedoidsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -197,7 +197,7 @@ Result<> ComputeKMedoidsFilter::executeImpl(DataStructure& dataStructure, const inputValues.ClusteringArrayPath = filterArgs.value(k_SelectedArrayPath_Key); inputValues.FeatureIdsArrayPath = inputValues.ClusteringArrayPath.replaceName(filterArgs.value(k_FeatureIdsArrayName_Key)); - return ComputeKMedoids(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeKMedoidsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp index 6da89db6e4..76d39de478 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeSurfaceAreaToVolumeFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -147,7 +147,7 @@ Result<> ComputeSurfaceAreaToVolumeFilter::executeImpl(DataStructure& dataStruct inputValues.SphericityArrayName = inputValues.NumCellsArrayPath.replaceName(filterArgs.value(k_SphericityArrayName_Key)); inputValues.InputImageGeometry = filterArgs.value(k_SelectedImageGeometryPath_Key); - return ComputeSurfaceAreaToVolume(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeSurfaceAreaToVolumeDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp index f9808f0c63..c2f4cc1672 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeSurfaceFeaturesFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -126,7 +126,7 @@ Result<> ComputeSurfaceFeaturesFilter::executeImpl(DataStructure& dataStructure, inputValues.InputImageGeometryPath = filterArgs.value(k_FeatureGeometryPath_Key); inputValues.MarkFeature0Neighbors = filterArgs.value(k_MarkFeature0Neighbors); inputValues.SurfaceFeaturesArrayName = filterArgs.value(k_SurfaceFeaturesArrayName_Key); - return ComputeSurfaceFeatures(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeSurfaceFeaturesDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp index c378d5a743..2ce7bc136b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp @@ -1,6 +1,6 @@ #include "DBSCANFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/DBSCAN.hpp" +#include "SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -69,7 +69,7 @@ Parameters DBSCANFilter::parameters() const // Create the parameter descriptors that are needed for this filter params.insertSeparator(Parameters::Separator{"Random Number Seed Parameters"}); params.insertLinkableParameter(std::make_unique( - k_ParseOrderIndex_Key, "Parse Order", "Whether to use random or low density first for parse order. See Documentation for further detail", to_underlying(DBSCAN::ParseOrder::LowDensityFirst), + k_ParseOrderIndex_Key, "Parse Order", "Whether to use random or low density first for parse order. See Documentation for further detail", to_underlying(DBSCANDirect::ParseOrder::LowDensityFirst), ChoicesParameter::Choices{"Low Density First", "Random", "Seeded Random"})); // sequence dependent DO NOT REORDER params.insert(std::make_unique>(k_SeedValue_Key, "Seed Value", "The seed fed into the random generator", std::mt19937::default_seed)); params.insert(std::make_unique(k_SeedArrayName_Key, "Stored Seed Value Array Name", "Name of array holding the seed value", "DBSCAN SeedValue")); @@ -100,9 +100,9 @@ Parameters DBSCANFilter::parameters() const std::make_unique(k_FeatureAMPath_Key, "Cluster Attribute Matrix", "The complete path to the attribute matrix in which to store to hold Cluster Data", DataPath{})); // Associate the Linkable Parameter(s) to the children parameters that they control - params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCAN::ParseOrder::Random))); - params.linkParameters(k_ParseOrderIndex_Key, k_SeedValue_Key, static_cast(to_underlying(DBSCAN::ParseOrder::SeededRandom))); - params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCAN::ParseOrder::SeededRandom))); + params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCANDirect::ParseOrder::Random))); + params.linkParameters(k_ParseOrderIndex_Key, k_SeedValue_Key, static_cast(to_underlying(DBSCANDirect::ParseOrder::SeededRandom))); + params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCANDirect::ParseOrder::SeededRandom))); params.linkParameters(k_UseMask_Key, k_MaskArrayPath_Key, true); return params; @@ -183,7 +183,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt } // For caching seed run to run - if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCAN::ParseOrder::LowDensityFirst) + if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCANDirect::ParseOrder::LowDensityFirst) { auto createAction = std::make_unique(DataType::uint64, std::vector{1}, std::vector{1}, DataPath({filterArgs.value(k_SeedArrayName_Key)})); resultOutputActions.value().appendAction(std::move(createAction)); @@ -204,12 +204,12 @@ Result<> DBSCANFilter::executeImpl(DataStructure& dataStructure, const Arguments } auto seed = filterArgs.value(k_SeedValue_Key); - if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCAN::ParseOrder::SeededRandom) + if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCANDirect::ParseOrder::SeededRandom) { seed = static_cast(std::chrono::steady_clock::now().time_since_epoch().count()); } - if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCAN::ParseOrder::LowDensityFirst) + if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCANDirect::ParseOrder::LowDensityFirst) { // Store Seed Value in Top Level Array dataStructure.getDataRefAs(DataPath({filterArgs.value(k_SeedArrayName_Key)}))[0] = seed; @@ -227,7 +227,7 @@ Result<> DBSCANFilter::executeImpl(DataStructure& dataStructure, const Arguments inputValues.ParseOrder = filterArgs.value(k_ParseOrderIndex_Key); inputValues.Seed = filterArgs.value(k_SeedValue_Key); - return DBSCAN(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return DBSCANDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp index fb5f642883..90fbf068bd 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp @@ -1,6 +1,6 @@ #include "FillBadDataFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/FillBadData.hpp" +#include "SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/DataPath.hpp" @@ -130,7 +130,7 @@ Result<> FillBadDataFilter::executeImpl(DataStructure& dataStructure, const Argu inputValues.ignoredDataArrayPaths = filterArgs.value(k_IgnoredDataArrayPaths_Key); inputValues.inputImageGeometry = filterArgs.value(k_SelectedImageGeometryPath_Key); - return FillBadData(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return FillBadDataBFS(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp index 0c705e6b60..86b63937ab 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp @@ -1,6 +1,6 @@ #include "IdentifySampleFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/IdentifySample.hpp" +#include "SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" @@ -110,7 +110,7 @@ Result<> IdentifySampleFilter::executeImpl(DataStructure& dataStructure, const A inputValues.InputImageGeometryPath = filterArgs.value(k_SelectedImageGeometryPath_Key); inputValues.MaskArrayPath = filterArgs.value(k_MaskArrayPath_Key); - return IdentifySample(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return IdentifySampleBFS(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp index 1834d2237d..4627a932ac 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp @@ -1,6 +1,6 @@ #include "MultiThresholdObjectsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp" +#include "SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -245,7 +245,7 @@ Result<> MultiThresholdObjectsFilter::executeImpl(DataStructure& dataStructure, inputValues.UseCustomFalseValue = filterArgs.value(k_UseCustomFalseValue); inputValues.UseCustomTrueValue = filterArgs.value(k_UseCustomTrueValue); - return MultiThresholdObjects(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return MultiThresholdObjectsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp index bc0fba90bc..5e58eadc79 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp @@ -1,6 +1,6 @@ #include "QuickSurfaceMeshFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp" +#include "SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/Filter/Actions/CopyArrayInstanceAction.hpp" @@ -235,7 +235,7 @@ Result<> QuickSurfaceMeshFilter::executeImpl(DataStructure& dataStructure, const } inputValues.CreatedDataArrayPaths = createdDataPaths; - return nx::core::QuickSurfaceMesh(dataStructure, &inputValues, shouldCancel, messageHandler)(); + return nx::core::QuickSurfaceMeshDirect(dataStructure, &inputValues, shouldCancel, messageHandler)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp index c9fb9862b7..41f3525f54 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp @@ -1,6 +1,6 @@ #include "SurfaceNetsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/SurfaceNets.hpp" +#include "SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" @@ -227,6 +227,6 @@ Result<> SurfaceNetsFilter::executeImpl(DataStructure& dataStructure, const Argu } inputValues.CreatedDataArrayPaths = createdDataPaths; - return SurfaceNets(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return SurfaceNetsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); } } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/test/DBSCANTest.cpp b/src/Plugins/SimplnxCore/test/DBSCANTest.cpp index 6d68ff413f..292270de70 100644 --- a/src/Plugins/SimplnxCore/test/DBSCANTest.cpp +++ b/src/Plugins/SimplnxCore/test/DBSCANTest.cpp @@ -4,7 +4,7 @@ #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" -#include "SimplnxCore/Filters/Algorithms/DBSCAN.hpp" +#include "SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp" #include "SimplnxCore/Filters/DBSCANFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" @@ -62,7 +62,7 @@ void LDFTestCase2D(const DataPath& targetPath, float32 epsilonVal, int32 minPtsV Arguments args; // Create default Parameters for the filter. - args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCAN::ParseOrder::LowDensityFirst))); + args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCANDirect::ParseOrder::LowDensityFirst))); args.insertOrAssign(DBSCANFilter::k_Epsilon_Key, std::make_any(epsilonVal)); args.insertOrAssign(DBSCANFilter::k_MinPoints_Key, std::make_any(minPtsVal)); args.insertOrAssign(DBSCANFilter::k_UseMask_Key, std::make_any(false)); @@ -109,7 +109,7 @@ std::vector BinPoints(const Int32Array& dataArray) void RandomTestCase2D(const DataPath& targetPath, float32 epsilonVal, int32 minPtsVal, const DataPath& exemplarClusterIds, ChoicesParameter::ValueType randomType) { - REQUIRE((randomType == to_underlying(DBSCAN::ParseOrder::Random) || randomType == to_underlying(DBSCAN::ParseOrder::SeededRandom))); + REQUIRE((randomType == to_underlying(DBSCANDirect::ParseOrder::Random) || randomType == to_underlying(DBSCANDirect::ParseOrder::SeededRandom))); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "dbscan_test.tar.gz", "dbscan_test"); DataStructure dataStructure = UnitTest::LoadDataStructure(k_2DTestFile); @@ -127,7 +127,7 @@ void RandomTestCase2D(const DataPath& targetPath, float32 epsilonVal, int32 minP // Create default Parameters for the filter. args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(randomType)); - args.insertOrAssign(DBSCANFilter::k_SeedValue_Key, std::make_any(k_Seed)); // Will be ignored if randomType == DBSCAN::ParseOrder::Random + args.insertOrAssign(DBSCANFilter::k_SeedValue_Key, std::make_any(k_Seed)); // Will be ignored if randomType == DBSCANDirect::ParseOrder::Random args.insertOrAssign(DBSCANFilter::k_SeedArrayName_Key, std::make_any("seed_array")); args.insertOrAssign(DBSCANFilter::k_Epsilon_Key, std::make_any(epsilonVal)); args.insertOrAssign(DBSCANFilter::k_MinPoints_Key, std::make_any(minPtsVal)); @@ -195,8 +195,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Aniso", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 4; // The exemplars were generated with LDF ::LDFTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath); - ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCAN::ParseOrder::Random); - ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCANDirect::ParseOrder::Random); + ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Blobs", "[SimplnxCore][DBSCAN]") @@ -205,8 +205,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Blobs", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath); - ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCAN::ParseOrder::Random); - ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCANDirect::ParseOrder::Random); + ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Circles", "[SimplnxCore][DBSCAN]") @@ -215,8 +215,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Circles", "[SimplnxCore][DBSCAN]" int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath); - ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCAN::ParseOrder::Random); - ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCANDirect::ParseOrder::Random); + ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Moons", "[SimplnxCore][DBSCAN]") @@ -225,8 +225,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Moons", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath); - ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCAN::ParseOrder::Random); - ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCANDirect::ParseOrder::Random); + ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: No Structure", "[SimplnxCore][DBSCAN]") @@ -235,8 +235,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: No Structure", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath); - ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCAN::ParseOrder::Random); - ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCANDirect::ParseOrder::Random); + ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Varied", "[SimplnxCore][DBSCAN]") @@ -245,8 +245,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Varied", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath); - ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCAN::ParseOrder::Random); - ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCANDirect::ParseOrder::Random); + ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 3D Test (LowDensityFirst)", "[SimplnxCore][DBSCAN]") @@ -268,7 +268,7 @@ TEST_CASE("SimplnxCore::DBSCAN: 3D Test (LowDensityFirst)", "[SimplnxCore][DBSCA Arguments args; // Create default Parameters for the filter. - args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCAN::ParseOrder::LowDensityFirst))); + args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCANDirect::ParseOrder::LowDensityFirst))); args.insertOrAssign(DBSCANFilter::k_Epsilon_Key, std::make_any(0.0099999998f)); args.insertOrAssign(DBSCANFilter::k_MinPoints_Key, std::make_any(5)); args.insertOrAssign(DBSCANFilter::k_UseMask_Key, std::make_any(false)); From ca022bb8f248ca085d32eb0b15019fbc3f31a1a1 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Wed, 8 Apr 2026 10:29:20 -0400 Subject: [PATCH 11/13] PERF: Out-of-core (OOC) optimized algorithms for SimplnxCore and OrientationAnalysis Replace per-element DataStore access with chunked bulk I/O (copyIntoBuffer/copyFromBuffer) across 60+ algorithm files to eliminate virtual dispatch overhead and HDF5 chunk thrashing when arrays are backed by out-of-core storage. --- Architecture --- DispatchAlgorithm pattern (Direct/Scanline): 11 algorithms gain a base dispatcher class that selects between an in-core Direct implementation and an OOC Scanline variant at runtime based on IsOutOfCore()/ForceOocAlgorithm(): SimplnxCore: ComputeBoundaryCells, ComputeFeatureNeighbors, ComputeKMedoids, ComputeSurfaceAreaToVolume, ComputeSurfaceFeatures, DBSCAN, MultiThresholdObjects, QuickSurfaceMesh, SurfaceNets OrientationAnalysis: BadDataNeighborOrientationCheck, ComputeIPFColors ComputeGBCDPoleFigure dispatches directly from its filter executeImpl(). Connected Component Labeling (CCL) pattern: 4 algorithms gain a two-pass CCL variant as an OOC alternative to random-access BFS/DFS flood-fill: SimplnxCore: FillBadData (BFS/CCL), IdentifySample (BFS/CCL) OrientationAnalysis: EBSDSegmentFeatures, CAxisSegmentFeatures The CCL engine in SegmentFeatures::executeCCL() scans voxels in Z-Y-X order with a 2-slice rolling buffer and UnionFind equivalence tracking, giving sequential I/O access patterns. Supports Face and FaceEdgeVertex connectivity with optional periodic boundaries. --- New utility infrastructure --- - UnionFind (src/simplnx/Utilities/UnionFind.hpp): Vector-based disjoint set with union-by-rank and path-halving. - SliceBufferedTransfer (src/simplnx/Utilities/SliceBufferedTransfer.hpp): Z-slice buffered tuple transfer for propagating neighbor voxel data used by ErodeDilate, FillBadData, MinNeighbors, and ReplaceElements. - TupleTransfer batch API (Filters/Algorithms/TupleTransfer.hpp): Batch bulk I/O methods for QuickSurfaceMesh and SurfaceNets mesh generation attribute transfer. - SegmentFeaturesTestUtils.hpp: Shared test builder functions for segmentation filter test suites. --- Bulk I/O conversions (existing algorithms) --- Core utilities: DataArrayUtilities (ImportFromBinaryFile, AppendData, CopyData, mirror ops), DataGroupUtilities (RemoveInactiveObjects), ClusteringUtilities (RandomizeFeatureIds), GeometryHelpers (FindElementsContainingVert, FindElementNeighbors), AlignSections (Z-slice OOC transfer path), ImageRotationUtilities (source slab caching for nearest-neighbor), TriangleUtilities (bulk-load triangles/labels for winding repair), H5DataStore (streaming row-batch FillOocDataStore replacing full- dataset allocation) SimplnxCore algorithms: AlignSectionsFeatureCentroid, ComputeEuclideanDistMap, ComputeFeatureCentroids, ComputeFeatureClustering, ComputeFeatureSizes, CropImageGeometry, ErodeDilateBadData, ErodeDilateCoordinationNumber, ErodeDilateMask, RegularGridSampleSurfaceMesh, RequireMinimumSizeFeatures, ReplaceElementAttributesWithNeighborValues, ScalarSegmentFeatures, WriteAvizoRectilinearCoordinate, WriteAvizoUniformCoordinate OrientationAnalysis algorithms: AlignSectionsMisorientation, AlignSectionsMutualInformation, ComputeAvgCAxes, ComputeAvgOrientations, ComputeCAxisLocations, ComputeFeatureNeighborCAxisMisalignments, ComputeFeatureReferenceCAxisMisorientations, ComputeFeatureReferenceMisorientations, ComputeGBCD, ComputeGBCDMetricBased, ComputeKernelAvgMisorientations, ComputeTwinBoundaries, ConvertOrientations, MergeTwins, NeighborOrientationCorrelation, RotateEulerRefFrame, WriteGBCDGMTFile, WriteGBCDTriangleData, WritePoleFigure EBSD readers: ReadAngData, ReadCtfData, ReadH5Ebsd, ReadH5EspritData --- Test infrastructure --- - UnitTestCommon: ExpectedStoreType()/RequireExpectedStoreType() helpers, TestFileSentinel reference-counted decompression, CompareDataArrays rewritten with chunked bulk I/O for OOC-safe comparison. - 29 test files updated with OOC dual-path testing: ForceOocAlgorithmGuard + GENERATE(from_range(k_ForceOocTestValues)) runs every test case in both in-core and forced-OOC modes. --- CMakeLists.txt | 1 + .../OrientationAnalysis/CMakeLists.txt | 7 +- .../AlignSectionsMisorientation.cpp | 373 ++++- .../AlignSectionsMisorientation.hpp | 11 +- .../AlignSectionsMutualInformation.cpp | 407 +++-- .../AlignSectionsMutualInformation.hpp | 27 +- .../BadDataNeighborOrientationCheck.cpp | 39 + .../BadDataNeighborOrientationCheck.hpp | 49 + ...adDataNeighborOrientationCheckScanline.cpp | 280 ++++ ...adDataNeighborOrientationCheckScanline.hpp | 35 + ...adDataNeighborOrientationCheckWorklist.cpp | 162 +- ...adDataNeighborOrientationCheckWorklist.hpp | 18 +- .../Algorithms/CAxisSegmentFeatures.cpp | 432 +++++- .../Algorithms/CAxisSegmentFeatures.hpp | 28 + .../Filters/Algorithms/ComputeAvgCAxes.cpp | 188 ++- .../Algorithms/ComputeAvgOrientations.cpp | 135 +- .../Algorithms/ComputeCAxisLocations.cpp | 70 +- ...mputeFeatureNeighborCAxisMisalignments.cpp | 57 +- ...teFeatureReferenceCAxisMisorientations.cpp | 98 +- ...ComputeFeatureReferenceMisorientations.cpp | 172 ++- .../Filters/Algorithms/ComputeGBCD.cpp | 128 +- .../Algorithms/ComputeGBCDMetricBased.cpp | 109 +- ...re.cpp => ComputeGBCDPoleFigureDirect.cpp} | 82 +- ...re.hpp => ComputeGBCDPoleFigureDirect.hpp} | 14 +- .../ComputeGBCDPoleFigureScanline.cpp | 309 ++++ .../ComputeGBCDPoleFigureScanline.hpp | 43 + .../Filters/Algorithms/ComputeIPFColors.cpp | 189 +-- .../Filters/Algorithms/ComputeIPFColors.hpp | 27 +- .../Algorithms/ComputeIPFColorsDirect.cpp | 206 +++ .../Algorithms/ComputeIPFColorsDirect.hpp | 38 + .../Algorithms/ComputeIPFColorsScanline.cpp | 166 +++ .../Algorithms/ComputeIPFColorsScanline.hpp | 34 + .../ComputeKernelAvgMisorientations.cpp | 297 ++-- .../Algorithms/ComputeTwinBoundaries.cpp | 196 ++- .../Algorithms/ConvertOrientations.cpp | 65 +- .../Algorithms/EBSDSegmentFeatures.cpp | 392 ++++- .../Algorithms/EBSDSegmentFeatures.hpp | 43 +- .../Filters/Algorithms/MergeTwins.cpp | 59 +- .../NeighborOrientationCorrelation.cpp | 300 ++-- .../NeighborOrientationCorrelation.hpp | 62 +- .../Filters/Algorithms/ReadAngData.cpp | 65 +- .../Filters/Algorithms/ReadCtfData.cpp | 96 +- .../Filters/Algorithms/ReadH5Ebsd.cpp | 103 +- .../Filters/Algorithms/ReadH5EspritData.cpp | 51 +- .../Algorithms/RotateEulerRefFrame.cpp | 118 +- .../Filters/Algorithms/WriteGBCDGMTFile.cpp | 32 +- .../Algorithms/WriteGBCDTriangleData.cpp | 79 +- .../Filters/Algorithms/WritePoleFigure.cpp | 328 ++-- .../BadDataNeighborOrientationCheckFilter.cpp | 4 +- .../Filters/ComputeGBCDPoleFigureFilter.cpp | 8 +- .../NeighborOrientationCorrelationFilter.hpp | 16 +- .../utilities/IEbsdOemReader.hpp | 10 +- .../test/AlignSectionsMisorientationTest.cpp | 141 +- .../AlignSectionsMutualInformationTest.cpp | 150 +- .../BadDataNeighborOrientationCheckTest.cpp | 336 +++++ .../test/CAxisSegmentFeaturesTest.cpp | 498 ++++--- .../test/ComputeGBCDPoleFigureTest.cpp | 4 + .../test/ComputeIPFColorsTest.cpp | 3 + .../test/EBSDSegmentFeaturesFilterTest.cpp | 499 ++++--- .../NeighborOrientationCorrelationTest.cpp | 175 +++ .../test/ReadH5EbsdTest.cpp | 8 +- .../test/RodriguesConvertorTest.cpp | 11 +- .../test/WritePoleFigureTest.cpp | 19 +- src/Plugins/SimplnxCore/CMakeLists.txt | 22 + .../AlignSectionsFeatureCentroid.cpp | 237 ++- .../AlignSectionsFeatureCentroid.hpp | 4 +- .../Algorithms/AppendImageGeometry.cpp | 3 +- .../Algorithms/ComputeArrayStatistics.cpp | 6 +- .../Algorithms/ComputeBoundaryCells.cpp | 34 + .../Algorithms/ComputeBoundaryCells.hpp | 46 + .../Algorithms/ComputeBoundaryCellsDirect.cpp | 17 +- .../Algorithms/ComputeBoundaryCellsDirect.hpp | 20 +- .../ComputeBoundaryCellsScanline.cpp | 177 +++ .../ComputeBoundaryCellsScanline.hpp | 39 + .../Algorithms/ComputeEuclideanDistMap.cpp | 218 ++- .../Algorithms/ComputeFeatureCentroids.cpp | 247 ++- .../Algorithms/ComputeFeatureClustering.cpp | 72 +- .../Algorithms/ComputeFeatureNeighbors.cpp | 29 + .../Algorithms/ComputeFeatureNeighbors.hpp | 56 + .../ComputeFeatureNeighborsDirect.cpp | 727 +++++++-- .../ComputeFeatureNeighborsDirect.hpp | 57 +- .../ComputeFeatureNeighborsScanline.cpp | 276 ++++ .../ComputeFeatureNeighborsScanline.hpp | 51 + .../Algorithms/ComputeFeatureSizes.cpp | 56 +- .../Filters/Algorithms/ComputeKMedoids.cpp | 41 + .../Filters/Algorithms/ComputeKMedoids.hpp | 49 + .../Algorithms/ComputeKMedoidsDirect.cpp | 16 +- .../Algorithms/ComputeKMedoidsDirect.hpp | 22 +- .../Algorithms/ComputeKMedoidsScanline.cpp | 277 ++++ .../Algorithms/ComputeKMedoidsScanline.hpp | 41 + .../Algorithms/ComputeSurfaceAreaToVolume.cpp | 35 + .../Algorithms/ComputeSurfaceAreaToVolume.hpp | 47 + .../ComputeSurfaceAreaToVolumeDirect.cpp | 54 +- .../ComputeSurfaceAreaToVolumeDirect.hpp | 22 +- .../ComputeSurfaceAreaToVolumeScanline.cpp | 203 +++ .../ComputeSurfaceAreaToVolumeScanline.hpp | 40 + .../Algorithms/ComputeSurfaceFeatures.cpp | 29 + .../Algorithms/ComputeSurfaceFeatures.hpp | 51 + .../ComputeSurfaceFeaturesDirect.cpp | 33 +- .../ComputeSurfaceFeaturesDirect.hpp | 23 +- .../ComputeSurfaceFeaturesScanline.cpp | 316 ++++ .../ComputeSurfaceFeaturesScanline.hpp | 39 + .../Filters/Algorithms/CropImageGeometry.cpp | 22 +- .../SimplnxCore/Filters/Algorithms/DBSCAN.cpp | 29 + .../SimplnxCore/Filters/Algorithms/DBSCAN.hpp | 58 + .../Filters/Algorithms/DBSCANDirect.cpp | 267 +--- .../Filters/Algorithms/DBSCANDirect.hpp | 32 +- .../Filters/Algorithms/DBSCANScanline.cpp | 1060 +++++++++++++ .../Filters/Algorithms/DBSCANScanline.hpp | 39 + .../Filters/Algorithms/ErodeDilateBadData.cpp | 230 +-- .../Filters/Algorithms/ErodeDilateBadData.hpp | 25 +- .../ErodeDilateCoordinationNumber.cpp | 209 ++- .../ErodeDilateCoordinationNumber.hpp | 24 +- .../Filters/Algorithms/ErodeDilateMask.cpp | 134 +- .../Filters/Algorithms/ErodeDilateMask.hpp | 24 +- .../Filters/Algorithms/FillBadData.cpp | 29 + .../Filters/Algorithms/FillBadData.hpp | 63 + .../Filters/Algorithms/FillBadDataBFS.cpp | 828 +++------- .../Filters/Algorithms/FillBadDataBFS.hpp | 136 +- .../Filters/Algorithms/FillBadDataCCL.cpp | 833 +++++++++++ .../Filters/Algorithms/FillBadDataCCL.hpp | 76 + .../Filters/Algorithms/IdentifySample.cpp | 29 + .../Filters/Algorithms/IdentifySample.hpp | 50 + .../Filters/Algorithms/IdentifySampleBFS.cpp | 324 +--- .../Filters/Algorithms/IdentifySampleBFS.hpp | 37 +- .../Filters/Algorithms/IdentifySampleCCL.cpp | 453 ++++++ .../Filters/Algorithms/IdentifySampleCCL.hpp | 61 + .../Algorithms/IdentifySampleCommon.hpp | 462 ++++++ .../Algorithms/MultiThresholdObjects.cpp | 35 + .../Algorithms/MultiThresholdObjects.hpp | 67 + .../MultiThresholdObjectsDirect.cpp | 79 +- .../MultiThresholdObjectsDirect.hpp | 39 +- .../MultiThresholdObjectsScanline.cpp | 285 ++++ .../MultiThresholdObjectsScanline.hpp | 39 + .../Filters/Algorithms/QuickSurfaceMesh.cpp | 27 + .../Filters/Algorithms/QuickSurfaceMesh.hpp | 62 + .../Algorithms/QuickSurfaceMeshDirect.cpp | 678 ++------- .../Algorithms/QuickSurfaceMeshDirect.hpp | 64 +- .../Algorithms/QuickSurfaceMeshScanline.cpp | 1326 +++++++++++++++++ .../Algorithms/QuickSurfaceMeshScanline.hpp | 56 + .../Filters/Algorithms/ReadHDF5Dataset.cpp | 11 + .../Filters/Algorithms/ReadStlFile.cpp | 6 +- .../RegularGridSampleSurfaceMesh.cpp | 111 +- .../RegularGridSampleSurfaceMesh.hpp | 21 +- ...aceElementAttributesWithNeighborValues.cpp | 184 ++- ...aceElementAttributesWithNeighborValues.hpp | 24 +- .../Algorithms/RequireMinimumSizeFeatures.cpp | 146 +- .../Algorithms/ScalarSegmentFeatures.cpp | 397 ++++- .../Algorithms/ScalarSegmentFeatures.hpp | 48 +- .../Filters/Algorithms/SurfaceNets.cpp | 33 + .../Filters/Algorithms/SurfaceNets.hpp | 58 + .../Filters/Algorithms/SurfaceNetsDirect.cpp | 69 +- .../Filters/Algorithms/SurfaceNetsDirect.hpp | 32 +- .../Algorithms/SurfaceNetsScanline.cpp | 893 +++++++++++ .../Algorithms/SurfaceNetsScanline.hpp | 60 + .../Filters/Algorithms/TupleTransfer.hpp | 317 ++++ .../WriteAvizoRectilinearCoordinate.cpp | 52 +- .../WriteAvizoUniformCoordinate.cpp | 44 +- .../Filters/ComputeBoundaryCellsFilter.cpp | 4 +- .../Filters/ComputeFeatureNeighborsFilter.cpp | 4 +- .../Filters/ComputeKMedoidsFilter.cpp | 4 +- .../ComputeSurfaceAreaToVolumeFilter.cpp | 4 +- .../Filters/ComputeSurfaceFeaturesFilter.cpp | 4 +- .../src/SimplnxCore/Filters/DBSCANFilter.cpp | 18 +- .../SimplnxCore/Filters/FillBadDataFilter.cpp | 4 +- .../Filters/IdentifySampleFilter.cpp | 4 +- .../Filters/MultiThresholdObjectsFilter.cpp | 4 +- .../Filters/QuickSurfaceMeshFilter.cpp | 4 +- .../SimplnxCore/Filters/SurfaceNetsFilter.cpp | 4 +- .../src/SimplnxCore/utils/VtkUtilities.hpp | 15 +- .../test/AlignSectionsFeatureCentroidTest.cpp | 129 +- .../test/AlignSectionsListTest.cpp | 123 ++ .../test/ComputeBoundaryCellsTest.cpp | 109 ++ .../test/ComputeFeatureNeighborsTest.cpp | 263 ++++ .../test/ComputeSurfaceAreaToVolumeTest.cpp | 132 ++ .../test/ComputeSurfaceFeaturesTest.cpp | 137 ++ src/Plugins/SimplnxCore/test/DBSCANTest.cpp | 34 +- .../test/ErodeDilateBadDataTest.cpp | 238 ++- .../ErodeDilateCoordinationNumberTest.cpp | 192 ++- .../SimplnxCore/test/ErodeDilateMaskTest.cpp | 205 ++- .../SimplnxCore/test/FillBadDataTest.cpp | 342 ++++- .../SimplnxCore/test/IdentifySampleTest.cpp | 241 ++- .../test/QuickSurfaceMeshFilterTest.cpp | 25 + .../SimplnxCore/test/ReadHDF5DatasetTest.cpp | 245 +-- .../SimplnxCore/test/ReadRawBinaryTest.cpp | 35 +- ...lementAttributesWithNeighborValuesTest.cpp | 237 ++- .../test/ScalarSegmentFeaturesTest.cpp | 391 +++-- .../SimplnxCore/test/SurfaceNetsTest.cpp | 29 + src/simplnx/Utilities/AlignSections.cpp | 123 +- src/simplnx/Utilities/ClusteringUtilities.cpp | 34 +- src/simplnx/Utilities/DataArrayUtilities.cpp | 23 +- src/simplnx/Utilities/DataArrayUtilities.hpp | 239 ++- src/simplnx/Utilities/DataGroupUtilities.cpp | 31 +- src/simplnx/Utilities/GeometryHelpers.hpp | 159 +- .../Utilities/ImageRotationUtilities.hpp | 209 ++- .../Utilities/Meshing/TriangleUtilities.cpp | 69 +- .../Utilities/Parsing/HDF5/H5DataStore.hpp | 84 +- src/simplnx/Utilities/SegmentFeatures.cpp | 888 ++++++++++- src/simplnx/Utilities/SegmentFeatures.hpp | 65 +- .../Utilities/SliceBufferedTransfer.hpp | 235 +++ src/simplnx/Utilities/UnionFind.hpp | 188 +++ .../UnitTest/SegmentFeaturesTestUtils.hpp | 632 ++++++++ .../simplnx/UnitTest/UnitTestCommon.hpp | 276 ++-- 203 files changed, 22869 insertions(+), 6197 deletions(-) create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp rename src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/{ComputeGBCDPoleFigure.cpp => ComputeGBCDPoleFigureDirect.cpp} (77%) rename src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/{ComputeGBCDPoleFigure.hpp => ComputeGBCDPoleFigureDirect.hpp} (60%) create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp create mode 100644 src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp create mode 100644 src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp create mode 100644 src/simplnx/Utilities/SliceBufferedTransfer.hpp create mode 100644 src/simplnx/Utilities/UnionFind.hpp create mode 100644 test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp diff --git a/CMakeLists.txt b/CMakeLists.txt index be6b83f810..815e96f45e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -572,6 +572,7 @@ set(SIMPLNX_HDRS ${SIMPLNX_SOURCE_DIR}/Utilities/SamplingUtils.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/SegmentFeatures.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/TimeUtilities.hpp + ${SIMPLNX_SOURCE_DIR}/Utilities/UnionFind.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/TooltipGenerator.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/TooltipRowItem.hpp ${SIMPLNX_SOURCE_DIR}/Utilities/OStreamUtilities.hpp diff --git a/src/Plugins/OrientationAnalysis/CMakeLists.txt b/src/Plugins/OrientationAnalysis/CMakeLists.txt index c2bbd6c43f..de37da3330 100644 --- a/src/Plugins/OrientationAnalysis/CMakeLists.txt +++ b/src/Plugins/OrientationAnalysis/CMakeLists.txt @@ -172,6 +172,8 @@ set(filter_algorithms AlignSectionsMisorientation AlignSectionsMutualInformation BadDataNeighborOrientationCheck + BadDataNeighborOrientationCheckScanline + BadDataNeighborOrientationCheckWorklist CAxisSegmentFeatures ComputeAvgCAxes ComputeAvgOrientations @@ -186,9 +188,12 @@ set(filter_algorithms ComputeFZQuaternions ComputeGBCD ComputeGBCDMetricBased - ComputeGBCDPoleFigure + ComputeGBCDPoleFigureDirect + ComputeGBCDPoleFigureScanline ComputeGBPDMetricBased ComputeIPFColors + ComputeIPFColorsDirect + ComputeIPFColorsScanline ComputeKernelAvgMisorientations ComputeMisorientations ComputeQuaternionConjugate diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp index 83a59b2db1..2050d915a0 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp @@ -3,13 +3,12 @@ #include "simplnx/Common/Numbers.hpp" #include "simplnx/DataStructure/DataGroup.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/MaskCompareUtilities.hpp" #include -#include - using namespace nx::core; // ----------------------------------------------------------------------------- @@ -39,8 +38,17 @@ Result<> AlignSectionsMisorientation::operator()() } // ----------------------------------------------------------------------------- -Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, std::vector& yShifts) +Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, std::vector& yShifts) { + { + const auto& quatsCheck = m_DataStructure.getDataRefAs(m_InputValues->quatsArrayPath); + const auto& cellPhasesCheck = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); + if(ForceOocAlgorithm() || IsOutOfCore(quatsCheck) || IsOutOfCore(cellPhasesCheck)) + { + return findShiftsOoc(xShifts, yShifts); + } + } + std::unique_ptr maskCompare = nullptr; if(m_InputValues->UseMask) { @@ -64,10 +72,10 @@ Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, SizeVec3 udims = gridGeom->getDimensions(); - std::array dims = { - static_cast(udims[0]), - static_cast(udims[1]), - static_cast(udims[2]), + std::array dims = { + static_cast(udims[0]), + static_cast(udims[1]), + static_cast(udims[2]), }; std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); @@ -75,10 +83,10 @@ Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, // Allocate a 2D Array which will be reused from slice to slice std::vector misorients(dims[0] * dims[1], false); - const auto halfDim0 = static_cast(dims[0] * 0.5f); - const auto halfDim1 = static_cast(dims[1] * 0.5f); + const auto halfDim0 = static_cast(dims[0] * 0.5f); + const auto halfDim1 = static_cast(dims[1] * 0.5f); - double deg2Rad = (nx::core::numbers::pi / 180.0); + float64 deg2Rad = (nx::core::numbers::pi / 180.0); ThrottledMessenger throttledMessenger = getMessageHelper().createThrottledMessenger(); if(m_InputValues->StoreAlignmentShifts) { @@ -86,64 +94,60 @@ Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, auto& relativeShiftsStore = m_DataStructure.getDataAs(m_InputValues->RelativeShiftsArrayPath)->getDataStoreRef(); auto& cumulativeShiftsStore = m_DataStructure.getDataAs(m_InputValues->CumulativeShiftsArrayPath)->getDataStoreRef(); // Loop over the Z Direction - for(int64_t iter = 1; iter < dims[2]; iter++) + for(int64 iter = 1; iter < dims[2]; iter++) { - if(m_ShouldCancel) - { - return {}; - } throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Determining Shifts || {:.2f}% Complete", CalculatePercentComplete(iter, dims[2])); }); if(getCancel()) { return {}; } - float minDisorientation = std::numeric_limits::max(); + float32 minDisorientation = std::numeric_limits::max(); // Work from the largest Slice Value to the lowest Slice Value. - int64_t slice = (dims[2] - 1) - iter; - int64_t oldxshift = -1; - int64_t oldyshift = -1; - int64_t newxshift = 0; - int64_t newyshift = 0; + int64 slice = (dims[2] - 1) - iter; + int64 oldxshift = -1; + int64 oldyshift = -1; + int64 newxshift = 0; + int64 newyshift = 0; // Initialize everything to false std::fill(misorients.begin(), misorients.end(), false); - float misorientationTolerance = static_cast(m_InputValues->misorientationTolerance * deg2Rad); + float32 misorientationTolerance = static_cast(m_InputValues->misorientationTolerance * deg2Rad); while(newxshift != oldxshift || newyshift != oldyshift) { oldxshift = newxshift; oldyshift = newyshift; - for(int32_t j = -3; j < 4; j++) + for(int32 j = -3; j < 4; j++) { - for(int32_t k = -3; k < 4; k++) + for(int32 k = -3; k < 4; k++) { - float disorientation = 0.0f; - float count = 0.0f; - int64_t xIdx = k + oldxshift + halfDim0; - int64_t yIdx = j + oldyshift + halfDim1; - int64_t idx = (dims[0] * yIdx) + xIdx; + float32 disorientation = 0.0f; + float32 count = 0.0f; + int64 xIdx = k + oldxshift + halfDim0; + int64 yIdx = j + oldyshift + halfDim1; + int64 idx = (dims[0] * yIdx) + xIdx; if(!misorients[idx] && llabs(k + oldxshift) < halfDim0 && llabs(j + oldyshift) < halfDim1) { - for(int64_t l = 0; l < dims[1]; l = l + 4) + for(int64 l = 0; l < dims[1]; l = l + 4) { - for(int64_t n = 0; n < dims[0]; n = n + 4) + for(int64 n = 0; n < dims[0]; n = n + 4) { if((l + j + oldyshift) >= 0 && (l + j + oldyshift) < dims[1] && (n + k + oldxshift) >= 0 && (n + k + oldxshift) < dims[0]) { count++; - int64_t refposition = ((slice + 1) * dims[0] * dims[1]) + (l * dims[0]) + n; - int64_t curposition = (slice * dims[0] * dims[1]) + ((l + j + oldyshift) * dims[0]) + (n + k + oldxshift); + int64 refposition = ((slice + 1) * dims[0] * dims[1]) + (l * dims[0]) + n; + int64 curposition = (slice * dims[0] * dims[1]) + ((l + j + oldyshift) * dims[0]) + (n + k + oldxshift); if(!m_InputValues->UseMask || maskCompare->bothTrue(refposition, curposition)) { - float angle = std::numeric_limits::max(); + float32 angle = std::numeric_limits::max(); if(cellPhases[refposition] > 0 && cellPhases[curposition] > 0) { ebsdlib::QuatD quat1(quats[refposition * 4], quats[refposition * 4 + 1], quats[refposition * 4 + 2], quats[refposition * 4 + 3]); // Makes a copy into voxQuat!!!! - auto laueClass1 = static_cast(crystalStructures[cellPhases[refposition]]); + auto laueClass1 = static_cast(crystalStructures[cellPhases[refposition]]); ebsdlib::QuatD quat2(quats[curposition * 4], quats[curposition * 4 + 1], quats[curposition * 4 + 2], quats[curposition * 4 + 3]); // Makes a copy into voxQuat!!!! - auto laueClass2 = static_cast(crystalStructures[cellPhases[curposition]]); - if(laueClass1 == laueClass2 && laueClass1 < static_cast(orientationOps.size())) + auto laueClass2 = static_cast(crystalStructures[cellPhases[curposition]]); + if(laueClass1 == laueClass2 && laueClass1 < static_cast(orientationOps.size())) { ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); angle = axisAngle[3]; @@ -198,60 +202,60 @@ Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, else { // Loop over the Z Direction - for(int64_t iter = 1; iter < dims[2]; iter++) + for(int64 iter = 1; iter < dims[2]; iter++) { throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Determining Shifts || {:.2f}% Complete", CalculatePercentComplete(iter, dims[2])); }); if(getCancel()) { return {}; } - float minDisorientation = std::numeric_limits::max(); + float32 minDisorientation = std::numeric_limits::max(); // Work from the largest Slice Value to the lowest Slice Value. - int64_t slice = (dims[2] - 1) - iter; - int64_t oldxshift = -1; - int64_t oldyshift = -1; - int64_t newxshift = 0; - int64_t newyshift = 0; + int64 slice = (dims[2] - 1) - iter; + int64 oldxshift = -1; + int64 oldyshift = -1; + int64 newxshift = 0; + int64 newyshift = 0; // Initialize everything to false std::fill(misorients.begin(), misorients.end(), false); - float misorientationTolerance = static_cast(m_InputValues->misorientationTolerance * deg2Rad); + float32 misorientationTolerance = static_cast(m_InputValues->misorientationTolerance * deg2Rad); while(newxshift != oldxshift || newyshift != oldyshift) { oldxshift = newxshift; oldyshift = newyshift; - for(int32_t j = -3; j < 4; j++) + for(int32 j = -3; j < 4; j++) { - for(int32_t k = -3; k < 4; k++) + for(int32 k = -3; k < 4; k++) { - float disorientation = 0.0f; - float count = 0.0f; - int64_t xIdx = k + oldxshift + halfDim0; - int64_t yIdx = j + oldyshift + halfDim1; - int64_t idx = (dims[0] * yIdx) + xIdx; + float32 disorientation = 0.0f; + float32 count = 0.0f; + int64 xIdx = k + oldxshift + halfDim0; + int64 yIdx = j + oldyshift + halfDim1; + int64 idx = (dims[0] * yIdx) + xIdx; if(!misorients[idx] && llabs(k + oldxshift) < halfDim0 && llabs(j + oldyshift) < halfDim1) { - for(int64_t l = 0; l < dims[1]; l = l + 4) + for(int64 l = 0; l < dims[1]; l = l + 4) { - for(int64_t n = 0; n < dims[0]; n = n + 4) + for(int64 n = 0; n < dims[0]; n = n + 4) { if((l + j + oldyshift) >= 0 && (l + j + oldyshift) < dims[1] && (n + k + oldxshift) >= 0 && (n + k + oldxshift) < dims[0]) { count++; - int64_t refposition = ((slice + 1) * dims[0] * dims[1]) + (l * dims[0]) + n; - int64_t curposition = (slice * dims[0] * dims[1]) + ((l + j + oldyshift) * dims[0]) + (n + k + oldxshift); + int64 refposition = ((slice + 1) * dims[0] * dims[1]) + (l * dims[0]) + n; + int64 curposition = (slice * dims[0] * dims[1]) + ((l + j + oldyshift) * dims[0]) + (n + k + oldxshift); if(!m_InputValues->UseMask || maskCompare->bothTrue(refposition, curposition)) { - float angle = std::numeric_limits::max(); + float32 angle = std::numeric_limits::max(); if(cellPhases[refposition] > 0 && cellPhases[curposition] > 0) { ebsdlib::QuatD quat1(quats[refposition * 4], quats[refposition * 4 + 1], quats[refposition * 4 + 2], quats[refposition * 4 + 3]); // Makes a copy into voxQuat!!!! - auto laueClass1 = static_cast(crystalStructures[cellPhases[refposition]]); + auto laueClass1 = static_cast(crystalStructures[cellPhases[refposition]]); ebsdlib::QuatD quat2(quats[curposition * 4], quats[curposition * 4 + 1], quats[curposition * 4 + 2], quats[curposition * 4 + 3]); // Makes a copy into voxQuat!!!! - auto laueClass2 = static_cast(crystalStructures[cellPhases[curposition]]); - if(laueClass1 == laueClass2 && laueClass1 < static_cast(orientationOps.size())) + auto laueClass2 = static_cast(crystalStructures[cellPhases[curposition]]); + if(laueClass1 == laueClass2 && laueClass1 < static_cast(orientationOps.size())) { ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); angle = axisAngle[3]; @@ -298,3 +302,252 @@ Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, return {}; } + +// ----------------------------------------------------------------------------- +// OOC-optimized findShifts: buffers 2 adjacent Z-slices of quats, cellPhases, +// and mask into local vectors before the convergence loop, eliminating random +// chunk-based DataStore access. +// ----------------------------------------------------------------------------- +Result<> AlignSectionsMisorientation::findShiftsOoc(std::vector& xShifts, std::vector& yShifts) +{ + // For OOC mask buffering, get the raw mask store for bulk reads instead of per-element isTrue() + const AbstractDataStore* maskUInt8StorePtr = nullptr; + const AbstractDataStore* maskBoolStorePtr = nullptr; + if(m_InputValues->UseMask) + { + const auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + if(maskArray.getDataType() == DataType::uint8) + { + maskUInt8StorePtr = &dynamic_cast&>(maskArray).getDataStoreRef(); + } + else if(maskArray.getDataType() == DataType::boolean) + { + maskBoolStorePtr = &dynamic_cast&>(maskArray).getDataStoreRef(); + } + else + { + return MakeErrorResult(-53900, fmt::format("Mask Array is not Bool or UInt8: {}", m_InputValues->MaskArrayPath.toString())); + } + } + + auto* gridGeom = m_DataStructure.getDataAs(m_InputValues->ImageGeometryPath); + + const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); + const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->quatsArrayPath); + const auto& crystalStructuresArray = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); + auto& cellPhasesStore = cellPhases.getDataStoreRef(); + auto& quatsStore = quats.getDataStoreRef(); + + // Cache ensemble-level array locally to avoid per-element virtual dispatch in hot loop + const auto& crystalStructuresStore = crystalStructuresArray.getDataStoreRef(); + std::vector crystalStructures(crystalStructuresStore.getSize()); + crystalStructuresStore.copyIntoBuffer(0, nonstd::span(crystalStructures.data(), crystalStructures.size())); + + SizeVec3 udims = gridGeom->getDimensions(); + + std::array dims = { + static_cast(udims[0]), + static_cast(udims[1]), + static_cast(udims[2]), + }; + + std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + + std::vector misorients(dims[0] * dims[1], false); + + const auto halfDim0 = static_cast(dims[0] * 0.5f); + const auto halfDim1 = static_cast(dims[1] * 0.5f); + + float64 deg2Rad = (nx::core::numbers::pi / 180.0); + + const int64 sliceVoxels = dims[0] * dims[1]; + + // Buffers for 2 Z-slices: reference (slice+1) and current (slice) + std::vector refQuatsBuf(sliceVoxels * 4); + std::vector curQuatsBuf(sliceVoxels * 4); + std::vector refPhasesBuf(sliceVoxels); + std::vector curPhasesBuf(sliceVoxels); + std::vector refMaskBuf; + std::vector curMaskBuf; + if(m_InputValues->UseMask) + { + refMaskBuf.resize(sliceVoxels, 1); + curMaskBuf.resize(sliceVoxels, 1); + } + + // Optional output stores + AbstractDataStore* slicesStorePtr = nullptr; + AbstractDataStore* relativeShiftsStorePtr = nullptr; + AbstractDataStore* cumulativeShiftsStorePtr = nullptr; + if(m_InputValues->StoreAlignmentShifts) + { + slicesStorePtr = &m_DataStructure.getDataAs(m_InputValues->SlicesArrayPath)->getDataStoreRef(); + relativeShiftsStorePtr = &m_DataStructure.getDataAs(m_InputValues->RelativeShiftsArrayPath)->getDataStoreRef(); + cumulativeShiftsStorePtr = &m_DataStructure.getDataAs(m_InputValues->CumulativeShiftsArrayPath)->getDataStoreRef(); + } + + // Pre-load the first reference slice (the top-most Z-slice) via bulk read + { + int64 firstRefOffset = (dims[2] - 1) * sliceVoxels; + cellPhasesStore.copyIntoBuffer(firstRefOffset, nonstd::span(refPhasesBuf.data(), sliceVoxels)); + quatsStore.copyIntoBuffer(firstRefOffset * 4, nonstd::span(refQuatsBuf.data(), sliceVoxels * 4)); + if(m_InputValues->UseMask) + { + if(maskUInt8StorePtr != nullptr) + { + maskUInt8StorePtr->copyIntoBuffer(firstRefOffset, nonstd::span(refMaskBuf.data(), sliceVoxels)); + } + else if(maskBoolStorePtr != nullptr) + { + auto boolBuf = std::make_unique(sliceVoxels); + maskBoolStorePtr->copyIntoBuffer(firstRefOffset, nonstd::span(boolBuf.get(), sliceVoxels)); + for(int64 idx = 0; idx < sliceVoxels; idx++) + { + refMaskBuf[idx] = boolBuf[idx] ? 1 : 0; + } + } + } + } + + for(int64 iter = 1; iter < dims[2]; iter++) + { + if(getCancel()) + { + return {}; + } + + int64 slice = (dims[2] - 1) - iter; + + // Bulk-read current slice (reference available from pre-load or previous iteration swap) + int64 curOffset = slice * sliceVoxels; + cellPhasesStore.copyIntoBuffer(curOffset, nonstd::span(curPhasesBuf.data(), sliceVoxels)); + quatsStore.copyIntoBuffer(curOffset * 4, nonstd::span(curQuatsBuf.data(), sliceVoxels * 4)); + if(m_InputValues->UseMask) + { + if(maskUInt8StorePtr != nullptr) + { + maskUInt8StorePtr->copyIntoBuffer(curOffset, nonstd::span(curMaskBuf.data(), sliceVoxels)); + } + else if(maskBoolStorePtr != nullptr) + { + auto boolBuf = std::make_unique(sliceVoxels); + maskBoolStorePtr->copyIntoBuffer(curOffset, nonstd::span(boolBuf.get(), sliceVoxels)); + for(int64 idx = 0; idx < sliceVoxels; idx++) + { + curMaskBuf[idx] = boolBuf[idx] ? 1 : 0; + } + } + } + + float32 minDisorientation = std::numeric_limits::max(); + int64 oldxshift = -1; + int64 oldyshift = -1; + int64 newxshift = 0; + int64 newyshift = 0; + + std::fill(misorients.begin(), misorients.end(), false); + + float32 misorientationTolerance = static_cast(m_InputValues->misorientationTolerance * deg2Rad); + + while(newxshift != oldxshift || newyshift != oldyshift) + { + oldxshift = newxshift; + oldyshift = newyshift; + for(int32 j = -3; j < 4; j++) + { + for(int32 k = -3; k < 4; k++) + { + float32 disorientation = 0.0f; + float32 count = 0.0f; + int64 xIdx = k + oldxshift + halfDim0; + int64 yIdx = j + oldyshift + halfDim1; + int64 idx = (dims[0] * yIdx) + xIdx; + if(!misorients[idx] && llabs(k + oldxshift) < halfDim0 && llabs(j + oldyshift) < halfDim1) + { + for(int64 l = 0; l < dims[1]; l = l + 4) + { + for(int64 n = 0; n < dims[0]; n = n + 4) + { + if((l + j + oldyshift) >= 0 && (l + j + oldyshift) < dims[1] && (n + k + oldxshift) >= 0 && (n + k + oldxshift) < dims[0]) + { + count++; + // Local buffer indices (within-slice) + int64 refLocalIdx = l * dims[0] + n; + int64 curLocalIdx = (l + j + oldyshift) * dims[0] + (n + k + oldxshift); + + bool maskOk = !m_InputValues->UseMask || (refMaskBuf[refLocalIdx] != 0 && curMaskBuf[curLocalIdx] != 0); + if(maskOk) + { + float32 angle = std::numeric_limits::max(); + if(refPhasesBuf[refLocalIdx] > 0 && curPhasesBuf[curLocalIdx] > 0) + { + ebsdlib::QuatD quat1(refQuatsBuf[refLocalIdx * 4], refQuatsBuf[refLocalIdx * 4 + 1], refQuatsBuf[refLocalIdx * 4 + 2], refQuatsBuf[refLocalIdx * 4 + 3]); + auto laueClass1 = static_cast(crystalStructures[refPhasesBuf[refLocalIdx]]); + ebsdlib::QuatD quat2(curQuatsBuf[curLocalIdx * 4], curQuatsBuf[curLocalIdx * 4 + 1], curQuatsBuf[curLocalIdx * 4 + 2], curQuatsBuf[curLocalIdx * 4 + 3]); + auto laueClass2 = static_cast(crystalStructures[curPhasesBuf[curLocalIdx]]); + if(laueClass1 == laueClass2 && laueClass1 < static_cast(orientationOps.size())) + { + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); + angle = axisAngle[3]; + } + } + if(angle > misorientationTolerance) + { + disorientation++; + } + } + if(m_InputValues->UseMask) + { + if(refMaskBuf[refLocalIdx] != 0 && curMaskBuf[curLocalIdx] == 0) + { + disorientation++; + } + if(refMaskBuf[refLocalIdx] == 0 && curMaskBuf[curLocalIdx] != 0) + { + disorientation++; + } + } + } + } + } + disorientation = disorientation / count; + xIdx = k + oldxshift + halfDim0; + yIdx = j + oldyshift + halfDim1; + idx = (dims[0] * yIdx) + xIdx; + misorients[idx] = true; + if(disorientation < minDisorientation || (disorientation == minDisorientation && ((llabs(k + oldxshift) < llabs(newxshift)) || (llabs(j + oldyshift) < llabs(newyshift))))) + { + newxshift = k + oldxshift; + newyshift = j + oldyshift; + minDisorientation = disorientation; + } + } + } + } + } + xShifts[iter] = xShifts[iter - 1] + newxshift; + yShifts[iter] = yShifts[iter - 1] + newyshift; + + if(m_InputValues->StoreAlignmentShifts) + { + usize xIndex = iter * 2; + usize yIndex = (iter * 2) + 1; + (*slicesStorePtr)[xIndex] = slice; + (*slicesStorePtr)[yIndex] = slice + 1; + (*relativeShiftsStorePtr)[xIndex] = newxshift; + (*relativeShiftsStorePtr)[yIndex] = newyshift; + (*cumulativeShiftsStorePtr)[xIndex] = xShifts[iter]; + (*cumulativeShiftsStorePtr)[yIndex] = yShifts[iter]; + } + + // Current slice becomes the reference for the next iteration (O(1) pointer swap) + std::swap(refQuatsBuf, curQuatsBuf); + std::swap(refPhasesBuf, curPhasesBuf); + if(m_InputValues->UseMask) + { + std::swap(refMaskBuf, curMaskBuf); + } + } + + return {}; +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp index 9db00a19d6..6da4a2c036 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp @@ -57,9 +57,18 @@ class ORIENTATIONANALYSIS_EXPORT AlignSectionsMisorientation : public AlignSecti * @param yShifts * @return Whether the x and y shifts were successfully found */ - Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; + Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; private: + /** + * @brief OOC-optimized variant of findShifts that buffers two adjacent Z-slices + * into local vectors before the convergence loop, eliminating per-tuple chunk thrashing. + * @param xShifts Output vector of cumulative X shifts per slice. + * @param yShifts Output vector of cumulative Y shifts per slice. + * @return Success or error result. + */ + Result<> findShiftsOoc(std::vector& xShifts, std::vector& yShifts); + DataStructure& m_DataStructure; const AlignSectionsMisorientationInputValues* m_InputValues = nullptr; const std::atomic_bool& m_ShouldCancel; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.cpp index 2df4c1ac5d..249ddb92d4 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.cpp @@ -5,6 +5,7 @@ #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/StringUtilities.hpp" @@ -41,25 +42,139 @@ Result<> AlignSectionsMutualInformation::operator()() } // ----------------------------------------------------------------------------- -Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, std::vector& yShifts) +int32 AlignSectionsMutualInformation::formFeaturesForSlice(const float32* quats, const int32* phases, const uint8* mask, std::vector& featureIds, int64 dimX, int64 dimY, + float32 misorientationTolerance, bool useMask, const std::vector& orientationOps, + const std::vector& crystalStructures) { - const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); - const AttributeMatrix* cellData = imageGeom.getCellData(); - auto totalPoints = static_cast(cellData->getNumberOfTuples()); - - if(m_InputValues->UseMask) + const int64 sliceVoxels = dimX * dimY; + usize initialVoxelsListSize = 1000; + std::vector voxelList(initialVoxelsListSize, -1); + int64 neighborPoints[4] = {-dimX, -1, 1, dimX}; + + int64 currentStartPoint = 0; + int32 featureCount = 1; + bool noSeeds = false; + while(!noSeeds) { - try + int64 seed = -1; + + for(int64 point = currentStartPoint; point < sliceVoxels; point++) + { + if((!useMask || (mask != nullptr && mask[point] != 0)) && featureIds[point] == 0 && phases[point] > 0) + { + seed = point; + currentStartPoint = point; + } + if(seed > -1) + { + break; + } + } + + if(seed == -1) { - m_MaskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); - } catch(const std::out_of_range& exception) + noSeeds = true; + } + if(seed >= 0) { - // This really should NOT be happening as the path was verified during preflight BUT we may be calling this from - // somewhere else that is NOT going through the normal nx::core::IFilter API of Preflight and Execute - std::string message = fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString()); - return MakeErrorResult(-53702, message); + std::vector::size_type size = 0; + featureIds[seed] = featureCount; + voxelList[size] = seed; + size++; + for(usize j = 0; j < size; ++j) + { + int64 currentpoint = voxelList[j]; + int64 col = currentpoint % dimX; + int64 row = currentpoint / dimX; + + auto q1Idx = currentpoint * 4; + ebsdlib::QuatD quat1(quats[q1Idx], quats[q1Idx + 1], quats[q1Idx + 2], quats[q1Idx + 3]); + uint32 laueClass1 = crystalStructures[phases[currentpoint]]; + for(int32 i = 0; i < 4; i++) + { + int64 neighbor = currentpoint + neighborPoints[i]; + if((i == 0) && row == 0) + { + continue; + } + if((i == 3) && row == (dimY - 1)) + { + continue; + } + if((i == 1) && col == 0) + { + continue; + } + if((i == 2) && col == (dimX - 1)) + { + continue; + } + if(featureIds[neighbor] <= 0 && phases[neighbor] > 0) + { + float32 angle = std::numeric_limits::max(); + auto q2Idx = neighbor * 4; + ebsdlib::QuatD quat2(quats[q2Idx], quats[q2Idx + 1], quats[q2Idx + 2], quats[q2Idx + 3]); + uint32 phase2 = crystalStructures[phases[neighbor]]; + + if(laueClass1 == phase2) + { + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); + angle = axisAngle[3]; + } + if(angle < misorientationTolerance) + { + featureIds[neighbor] = featureCount; + voxelList[size] = neighbor; + size++; + if(size >= voxelList.size()) + { + size = voxelList.size(); + voxelList.resize(size + initialVoxelsListSize); + for(std::vector::size_type v = size; v < voxelList.size(); ++v) + { + voxelList[v] = -1; + } + } + } + } + } + } + voxelList.erase(std::remove(voxelList.begin(), voxelList.end(), -1), voxelList.end()); + featureCount++; + voxelList.assign(initialVoxelsListSize, -1); } } + return featureCount; +} + +namespace +{ +/** + * @brief Helper to buffer one slice of mask data into a uint8 vector. + */ +void bufferMaskSlice(const AbstractDataStore* maskUInt8StorePtr, const AbstractDataStore* maskBoolStorePtr, int64 sliceOffset, int64 sliceVoxels, std::vector& maskBuf) +{ + if(maskUInt8StorePtr != nullptr) + { + maskUInt8StorePtr->copyIntoBuffer(sliceOffset, nonstd::span(maskBuf.data(), sliceVoxels)); + } + else if(maskBoolStorePtr != nullptr) + { + auto boolBuf = std::make_unique(sliceVoxels); + maskBoolStorePtr->copyIntoBuffer(sliceOffset, nonstd::span(boolBuf.get(), sliceVoxels)); + for(int64 idx = 0; idx < sliceVoxels; idx++) + { + maskBuf[idx] = boolBuf[idx] ? 1 : 0; + } + } +} + +} // namespace + +// ----------------------------------------------------------------------------- +Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, std::vector& yShifts) +{ + const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); SizeVec3 udims = imageGeom.getDimensions(); int64 dims[3] = { @@ -68,16 +183,85 @@ Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, static_cast(udims[2]), }; - std::vector miFeatureIds(totalPoints, 0); - std::vector featureCounts(dims[2], 0); + const int64 sliceVoxels = dims[0] * dims[1]; + + // Set up orientation ops and crystal structures + auto orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + const auto& crystalStructuresArray = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + const auto& crystalStructuresStore = crystalStructuresArray.getDataStoreRef(); + std::vector crystalStructures(crystalStructuresStore.getSize()); + crystalStructuresStore.copyIntoBuffer(0, nonstd::span(crystalStructures.data(), crystalStructures.size())); + + float32 misorientationTolerance = m_InputValues->MisorientationTolerance * nx::core::Constants::k_PiOver180F; + + // Get store refs for bulk reads (copyIntoBuffer works for both in-core and OOC) + const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); + const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); + auto& quatsStore = quats.getDataStoreRef(); + auto& cellPhasesStore = cellPhases.getDataStoreRef(); + + // For bulk mask reads + const AbstractDataStore* maskUInt8StorePtr = nullptr; + const AbstractDataStore* maskBoolStorePtr = nullptr; + if(m_InputValues->UseMask) + { + const auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + if(maskArray.getDataType() == DataType::uint8) + { + maskUInt8StorePtr = &dynamic_cast&>(maskArray).getDataStoreRef(); + } + else if(maskArray.getDataType() == DataType::boolean) + { + maskBoolStorePtr = &dynamic_cast&>(maskArray).getDataStoreRef(); + } + } + + // Rolling buffers for 2-slice approach + std::vector refFeatureIds(sliceVoxels, 0); + std::vector curFeatureIds(sliceVoxels, 0); + int32 refFeatureCount = 0; + int32 curFeatureCount = 0; + + // Per-slice buffers for bulk reads + std::vector quatsBuf(sliceVoxels * 4); + std::vector phasesBuf(sliceVoxels); + std::vector maskBuf; + if(m_InputValues->UseMask) + { + maskBuf.resize(sliceVoxels, 1); + } + + // Lambda to flood-fill a single slice into a feature ID buffer, returning the feature count + auto floodFillSlice = [&](int64 sliceIndex, std::vector& featureIds) -> int32 { + std::fill(featureIds.begin(), featureIds.end(), 0); + + int64 sliceOffset = sliceIndex * sliceVoxels; + + // Bulk-read this slice's data into local buffers + cellPhasesStore.copyIntoBuffer(sliceOffset, nonstd::span(phasesBuf.data(), sliceVoxels)); + quatsStore.copyIntoBuffer(sliceOffset * 4, nonstd::span(quatsBuf.data(), sliceVoxels * 4)); + + const uint8* sliceMask = nullptr; + if(m_InputValues->UseMask) + { + bufferMaskSlice(maskUInt8StorePtr, maskBoolStorePtr, sliceOffset, sliceVoxels, maskBuf); + sliceMask = maskBuf.data(); + } + + return formFeaturesForSlice(quatsBuf.data(), phasesBuf.data(), sliceMask, featureIds, dims[0], dims[1], misorientationTolerance, m_InputValues->UseMask, orientationOps, crystalStructures); + }; + + // Pre-flood-fill the topmost slice (slice = dims[2]-1) into refFeatureIds. + // The iteration goes from iter=1..dims[2]-1, where slice = (dims[2]-1) - iter. + // The reference slice is slice+1; for iter=1, that's slice+1 = dims[2]-1. + int64 topSlice = dims[2] - 1; + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Identifying Features: Slice {}/{} complete", topSlice, dims[2])); + refFeatureCount = floodFillSlice(topSlice, refFeatureIds); std::vector> mutualInfo12; std::vector mutualInfo1; std::vector mutualInfo2; - // Segment each slice - formFeaturesSections(miFeatureIds, featureCounts); - std::vector> misorientations(dims[0]); for(int64 i = 0; i < dims[0]; i++) { @@ -95,16 +279,22 @@ Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, { return {}; } - m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Determining Shifts: Slice {}/{} complete", iter, dims[2])); - float32 minDisorientation = std::numeric_limits::max(); int64 slice = (dims[2] - 1) - iter; - int32 featureCount1 = featureCounts[slice]; - int32 featureCount2 = featureCounts[slice + 1]; + + // Flood-fill the current slice + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Identifying Features: Slice {}/{} complete", slice, dims[2])); + curFeatureCount = floodFillSlice(slice, curFeatureIds); + + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Determining Shifts: Slice {}/{} complete", iter, dims[2])); + + int32 featureCount1 = curFeatureCount; + int32 featureCount2 = refFeatureCount; mutualInfo12 = std::vector>(featureCount1, std::vector(featureCount2, 0.0f)); mutualInfo1 = std::vector(featureCount1, 0.0f); mutualInfo2 = std::vector(featureCount2, 0.0f); + float32 minDisorientation = std::numeric_limits::max(); int64 oldXShift = -1; int64 oldYShift = -1; int64 newXShift = 0; @@ -134,10 +324,10 @@ Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, { if((dim1Index + j + oldYShift) >= 0 && (dim1Index + j + oldYShift) < dims[1] && (dim0Index + k + oldXShift) >= 0 && (dim0Index + k + oldXShift) < dims[0]) { - int64 refPosition = ((slice + 1) * dims[0] * dims[1]) + (dim1Index * dims[0]) + dim0Index; - int64 curPosition = (slice * dims[0] * dims[1]) + ((dim1Index + j + oldYShift) * dims[0]) + (dim0Index + k + oldXShift); - int32 refGNum = miFeatureIds[refPosition]; - int32 curGNum = miFeatureIds[curPosition]; + int64 refLocalIdx = dim1Index * dims[0] + dim0Index; + int64 curLocalIdx = (dim1Index + j + oldYShift) * dims[0] + (dim0Index + k + oldXShift); + int32 refGNum = refFeatureIds[refLocalIdx]; + int32 curGNum = curFeatureIds[curLocalIdx]; if(curGNum >= 0 && refGNum >= 0) { mutualInfo12[curGNum][refGNum]++; @@ -211,22 +401,36 @@ Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, relativeShiftsStore[yIndex] = newYShift; cumulativeShiftsStore[xIndex] = xShifts[iter]; cumulativeShiftsStore[yIndex] = yShifts[iter]; + + // Roll: current slice becomes reference for the next iteration + std::swap(refFeatureIds, curFeatureIds); + refFeatureCount = curFeatureCount; } } else { for(int64 iter = 1; iter < dims[2]; iter++) { - m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Determining Shifts: Slice {}/{} complete", iter, dims[2])); + if(m_ShouldCancel) + { + return {}; + } - float32 minDisorientation = std::numeric_limits::max(); int64 slice = (dims[2] - 1) - iter; - int32 featureCount1 = featureCounts[slice]; - int32 featureCount2 = featureCounts[slice + 1]; + + // Flood-fill the current slice + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Identifying Features: Slice {}/{} complete", slice, dims[2])); + curFeatureCount = floodFillSlice(slice, curFeatureIds); + + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Determining Shifts: Slice {}/{} complete", iter, dims[2])); + + int32 featureCount1 = curFeatureCount; + int32 featureCount2 = refFeatureCount; mutualInfo12 = std::vector>(featureCount1, std::vector(featureCount2, 0.0f)); mutualInfo1 = std::vector(featureCount1, 0.0f); mutualInfo2 = std::vector(featureCount2, 0.0f); + float32 minDisorientation = std::numeric_limits::max(); int64 oldXShift = -1; int64 oldYShift = -1; int64 newXShift = 0; @@ -256,10 +460,10 @@ Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, { if((dim1Index + j + oldYShift) >= 0 && (dim1Index + j + oldYShift) < dims[1] && (dim0Index + k + oldXShift) >= 0 && (dim0Index + k + oldXShift) < dims[0]) { - int64 refPosition = ((slice + 1) * dims[0] * dims[1]) + (dim1Index * dims[0]) + dim0Index; - int64 curPosition = (slice * dims[0] * dims[1]) + ((dim1Index + j + oldYShift) * dims[0]) + (dim0Index + k + oldXShift); - int32 refGNum = miFeatureIds[refPosition]; - int32 curGNum = miFeatureIds[curPosition]; + int64 refLocalIdx = dim1Index * dims[0] + dim0Index; + int64 curLocalIdx = (dim1Index + j + oldYShift) * dims[0] + (dim0Index + k + oldXShift); + int32 refGNum = refFeatureIds[refLocalIdx]; + int32 curGNum = curFeatureIds[curLocalIdx]; if(curGNum >= 0 && refGNum >= 0) { mutualInfo12[curGNum][refGNum]++; @@ -324,139 +528,12 @@ Result<> AlignSectionsMutualInformation::findShifts(std::vector& xShifts, } xShifts[iter] = xShifts[iter - 1] + newXShift; yShifts[iter] = yShifts[iter - 1] + newYShift; + + // Roll: current slice becomes reference for the next iteration + std::swap(refFeatureIds, curFeatureIds); + refFeatureCount = curFeatureCount; } } return {}; } - -// ----------------------------------------------------------------------------- -void AlignSectionsMutualInformation::formFeaturesSections(std::vector& miFeatureIds, std::vector& featureCounts) -{ - const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); - - SizeVec3 udims = imageGeom.getDimensions(); - int64 dims[3] = { - static_cast(udims[0]), - static_cast(udims[1]), - static_cast(udims[2]), - }; - - auto orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - - auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); - auto& m_CellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); - auto& m_CrystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); - - size_t initialVoxelsListSize = 1000; - - float misorientationTolerance = m_InputValues->MisorientationTolerance * nx::core::Constants::k_PiOver180F; - - featureCounts.resize(dims[2]); - - std::vector voxelList(initialVoxelsListSize, -1); - int64_t neighborPoints[4] = {-dims[0], -1, 1, dims[0]}; - - for(int64_t slice = 0; slice < dims[2]; slice++) - { - m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Identifying Features: Slice {}/{} complete", slice, dims[2])); - - int64 startPoint = slice * dims[0] * dims[1]; - int64 endPoint = (slice + 1) * dims[0] * dims[1]; - int64 currentStartPoint = startPoint; - - int32 featureCount = 1; - bool noSeeds = false; - while(!noSeeds) - { - int64 seed = -1; - - for(int64 point = currentStartPoint; point < endPoint; point++) - { - if((!m_InputValues->UseMask || (m_MaskCompare != nullptr && m_MaskCompare->isTrue(point))) && miFeatureIds[point] == 0 && m_CellPhases[point] > 0) - { - seed = point; - currentStartPoint = point; - } - if(seed > -1) - { - break; - } - } - - if(seed == -1) - { - noSeeds = true; - } - if(seed >= 0) - { - std::vector::size_type size = 0; - miFeatureIds[seed] = featureCount; - voxelList[size] = seed; - size++; - for(size_t j = 0; j < size; ++j) - { - int64_t currentpoint = voxelList[j]; - int64 col = currentpoint % dims[0]; - int64 row = (currentpoint / dims[0]) % dims[1]; - - auto q1TupleIndex = currentpoint * 4; - ebsdlib::QuatD quat1(quats[q1TupleIndex], quats[q1TupleIndex + 1], quats[q1TupleIndex + 2], quats[q1TupleIndex + 3]); - uint32_t laueClass1 = m_CrystalStructures[m_CellPhases[currentpoint]]; - for(int32_t i = 0; i < 4; i++) - { - int64 neighbor = currentpoint + neighborPoints[i]; - if((i == 0) && row == 0) - { - continue; - } - if((i == 3) && row == (dims[1] - 1)) - { - continue; - } - if((i == 1) && col == 0) - { - continue; - } - if((i == 2) && col == (dims[0] - 1)) - { - continue; - } - if(miFeatureIds[neighbor] <= 0 && m_CellPhases[neighbor] > 0) - { - float32 angle = std::numeric_limits::max(); - auto q2TupleIndex = neighbor * 4; - ebsdlib::QuatD quat2(quats[q2TupleIndex], quats[q2TupleIndex + 1], quats[q2TupleIndex + 2], quats[q2TupleIndex + 3]); - uint32_t phase2 = m_CrystalStructures[m_CellPhases[neighbor]]; - - if(laueClass1 == phase2) - { - ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); - angle = axisAngle[3]; - } - if(angle < misorientationTolerance) - { - miFeatureIds[neighbor] = featureCount; - voxelList[size] = neighbor; - size++; - if(size >= voxelList.size()) - { - size = voxelList.size(); - voxelList.resize(size + initialVoxelsListSize); - for(std::vector::size_type v = size; v < voxelList.size(); ++v) - { - voxelList[v] = -1; - } - } - } - } - } - } - voxelList.erase(std::remove(voxelList.begin(), voxelList.end(), -1), voxelList.end()); - featureCount++; - voxelList.assign(initialVoxelsListSize, -1); - } - } - featureCounts[slice] = featureCount; - } -} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp index 224d254b4f..1a82433041 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp @@ -9,7 +9,8 @@ #include "simplnx/Parameters/FileSystemPathParameter.hpp" #include "simplnx/Parameters/NumberParameter.hpp" #include "simplnx/Utilities/AlignSections.hpp" -#include "simplnx/Utilities/MaskCompareUtilities.hpp" + +#include namespace nx::core { @@ -51,15 +52,31 @@ class ORIENTATIONANALYSIS_EXPORT AlignSectionsMutualInformation : public AlignSe protected: Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; - void formFeaturesSections(std::vector& miFeatureIds, std::vector& featureCounts); - private: + /** + * @brief Flood-fills a single Z-slice to identify features based on + * misorientation tolerance. Uses local indices (0 to sliceVoxels-1). + * Works for both in-core and OOC paths since the caller provides + * pre-buffered slice data. + * @param quats Pointer to the slice's quaternion data (4 components per voxel). + * @param phases Pointer to the slice's phase data. + * @param mask Pointer to the slice's mask data (nullptr if mask is not used). + * @param featureIds Output vector of per-voxel feature IDs (must be pre-zeroed, size = dimX*dimY). + * @param dimX Number of voxels in X dimension. + * @param dimY Number of voxels in Y dimension. + * @param misorientationTolerance Misorientation tolerance in radians. + * @param useMask Whether to use the mask array. + * @param orientationOps Laue orientation operators. + * @param crystalStructures Crystal structure IDs indexed by phase. + * @return The feature count for this slice. + */ + int32 formFeaturesForSlice(const float32* quats, const int32* phases, const uint8* mask, std::vector& featureIds, int64 dimX, int64 dimY, float32 misorientationTolerance, bool useMask, + const std::vector& orientationOps, const std::vector& crystalStructures); + DataStructure& m_DataStructure; const AlignSectionsMutualInformationInputValues* m_InputValues = nullptr; const std::atomic_bool& m_ShouldCancel; const IFilter::MessageHandler& m_MessageHandler; - - std::unique_ptr m_MaskCompare = nullptr; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp new file mode 100644 index 0000000000..92bf066b3c --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp @@ -0,0 +1,39 @@ +#include "BadDataNeighborOrientationCheck.hpp" + +#include "BadDataNeighborOrientationCheckScanline.hpp" +#include "BadDataNeighborOrientationCheckWorklist.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +BadDataNeighborOrientationCheck::BadDataNeighborOrientationCheck(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + BadDataNeighborOrientationCheckInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +BadDataNeighborOrientationCheck::~BadDataNeighborOrientationCheck() noexcept = default; + +// ----------------------------------------------------------------------------- +const std::atomic_bool& BadDataNeighborOrientationCheck::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> BadDataNeighborOrientationCheck::operator()() +{ + auto* quatsArray = m_DataStructure.getDataAs(m_InputValues->QuatsArrayPath); + auto* maskArray = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); + auto* phasesArray = m_DataStructure.getDataAs(m_InputValues->CellPhasesArrayPath); + + return DispatchAlgorithm({quatsArray, maskArray, phasesArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, + m_InputValues); +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp new file mode 100644 index 0000000000..5a65acab3d --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp @@ -0,0 +1,49 @@ +#pragma once + +#include "OrientationAnalysis/OrientationAnalysis_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckInputValues +{ + float32 MisorientationTolerance; + int32 NumberOfNeighbors; + DataPath ImageGeomPath; + DataPath QuatsArrayPath; + DataPath MaskArrayPath; + DataPath CellPhasesArrayPath; + DataPath CrystalStructuresArrayPath; +}; + +/** + * @class + */ +class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheck +{ +public: + BadDataNeighborOrientationCheck(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + BadDataNeighborOrientationCheckInputValues* inputValues); + ~BadDataNeighborOrientationCheck() noexcept; + + BadDataNeighborOrientationCheck(const BadDataNeighborOrientationCheck&) = delete; + BadDataNeighborOrientationCheck(BadDataNeighborOrientationCheck&&) noexcept = delete; + BadDataNeighborOrientationCheck& operator=(const BadDataNeighborOrientationCheck&) = delete; + BadDataNeighborOrientationCheck& operator=(BadDataNeighborOrientationCheck&&) noexcept = delete; + + Result<> operator()(); + + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp new file mode 100644 index 0000000000..f715c2daa7 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp @@ -0,0 +1,280 @@ +#include "BadDataNeighborOrientationCheckScanline.hpp" + +#include "BadDataNeighborOrientationCheck.hpp" + +#include "simplnx/Common/Numbers.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/MaskCompareUtilities.hpp" + +#include + +#include + +using namespace nx::core; + +namespace +{ +/** + * @brief Computes misorientation between quat1 and the neighbor at neighborSliceIdx. + * Returns true if same phase, phase > 0, and misorientation < tolerance. + */ +inline bool isMisorientationMatch(int64 neighborSliceIdx, const std::vector& neighborQuats, const std::vector& neighborPhases, int32 curPhase, uint32 laueClass, + const ebsdlib::QuatD& quat1, float32 misorientationTolerance, const std::vector& orientationOps) +{ + const int32 neighborPhase = neighborPhases[neighborSliceIdx]; + if(curPhase != neighborPhase || curPhase <= 0) + { + return false; + } + const int64 nqOffset = neighborSliceIdx * 4; + ebsdlib::QuatD quat2(neighborQuats[nqOffset], neighborQuats[nqOffset + 1], neighborQuats[nqOffset + 2], neighborQuats[nqOffset + 3]); + quat2.positiveOrientation(); + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass]->calculateMisorientation(quat1, quat2); + return axisAngle[3] < misorientationTolerance; +} + +/** + * @brief Counts matching good face-neighbors for a bad voxel at (xIdx, yIdx) + * within the current Z-slice, using 3-slice rolling window buffers. + * Returns the number of good neighbors with matching orientation. + */ +inline int32 countMatchingNeighbors(int64 xIdx, int64 yIdx, int64 zIdx, int64 dimX, int64 dimY, int64 dimZ, int64 sliceIndex, const std::vector& prevQuats, + const std::vector& curQuats, const std::vector& nextQuats, const std::vector& prevPhases, const std::vector& curPhases, + const std::vector& nextPhases, const std::vector& prevMask, const std::vector& curMask, const std::vector& nextMask, int32 curPhase, + uint32 laueClass, const ebsdlib::QuatD& quat1, float32 misorientationTolerance, const std::vector& orientationOps) +{ + int32 count = 0; + if(xIdx > 0 && curMask[sliceIndex - 1] && isMisorientationMatch(sliceIndex - 1, curQuats, curPhases, curPhase, laueClass, quat1, misorientationTolerance, orientationOps)) + { + count++; + } + if(xIdx < dimX - 1 && curMask[sliceIndex + 1] && isMisorientationMatch(sliceIndex + 1, curQuats, curPhases, curPhase, laueClass, quat1, misorientationTolerance, orientationOps)) + { + count++; + } + if(yIdx > 0 && curMask[sliceIndex - dimX] && isMisorientationMatch(sliceIndex - dimX, curQuats, curPhases, curPhase, laueClass, quat1, misorientationTolerance, orientationOps)) + { + count++; + } + if(yIdx < dimY - 1 && curMask[sliceIndex + dimX] && isMisorientationMatch(sliceIndex + dimX, curQuats, curPhases, curPhase, laueClass, quat1, misorientationTolerance, orientationOps)) + { + count++; + } + if(zIdx > 0 && prevMask[sliceIndex] && isMisorientationMatch(sliceIndex, prevQuats, prevPhases, curPhase, laueClass, quat1, misorientationTolerance, orientationOps)) + { + count++; + } + if(zIdx < dimZ - 1 && nextMask[sliceIndex] && isMisorientationMatch(sliceIndex, nextQuats, nextPhases, curPhase, laueClass, quat1, misorientationTolerance, orientationOps)) + { + count++; + } + return count; +} +} // namespace + +// ----------------------------------------------------------------------------- +BadDataNeighborOrientationCheckScanline::BadDataNeighborOrientationCheckScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const BadDataNeighborOrientationCheckInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +BadDataNeighborOrientationCheckScanline::~BadDataNeighborOrientationCheckScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief Flips bad voxels to good using Z-slice rolling window bulk I/O. + * + * Zero O(N) memory: no per-voxel neighborCount or mask arrays. Instead, + * neighbor counts are recomputed on-the-fly for each bad voxel during every + * pass using only O(slice) rolling window buffers. The mask is read/written + * per-slice from the real OOC-backed mask array. + * + * For each level (6 down to NumberOfNeighbors), repeatedly scan the volume. + * For each bad voxel, recompute the count of matching good face-neighbors. + * If count >= currentLevel, flip the voxel. Repeat until no flips occur. + */ +Result<> BadDataNeighborOrientationCheckScanline::operator()() +{ + const float32 misorientationTolerance = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; + + const auto* imageGeomPtr = m_DataStructure.getDataAs(m_InputValues->ImageGeomPath); + SizeVec3 udims = imageGeomPtr->getDimensions(); + const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); + auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); + const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + + std::unique_ptr maskCompare; + try + { + maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); + } catch(const std::out_of_range& exception) + { + return MakeErrorResult(-54900, fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString())); + } + + const int64 dimX = static_cast(udims[0]); + const int64 dimY = static_cast(udims[1]); + const int64 dimZ = static_cast(udims[2]); + const int64 xyStride = dimX * dimY; + const usize sliceSize = static_cast(dimY) * static_cast(dimX); + const usize quatSliceElems = sliceSize * 4; + + std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + + // Cache ensemble-level CrystalStructures locally (tiny array) + const usize numCrystalStructures = crystalStructures.getNumberOfTuples(); + std::vector localCrystalStructures(numCrystalStructures); + { + const auto& csStore = crystalStructures.getDataStoreRef(); + csStore.copyIntoBuffer(0, nonstd::span(localCrystalStructures.data(), numCrystalStructures)); + } + + auto& quatsStore = quats.getDataStoreRef(); + const auto& phasesStore = cellPhases.getDataStoreRef(); + + // Get the mask DataStore for bulk slice reads and per-element flip writes + auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + const bool maskIsUInt8 = (maskArray.getDataType() == DataType::uint8); + AbstractDataStore* maskStorePtr = nullptr; + if(maskIsUInt8) + { + maskStorePtr = &dynamic_cast(maskArray).getDataStoreRef(); + } + + // Rolling window buffers — O(slice) memory only + std::vector prevQuats(quatSliceElems); + std::vector curQuats(quatSliceElems); + std::vector nextQuats(quatSliceElems); + std::vector prevPhases(sliceSize); + std::vector curPhases(sliceSize); + std::vector nextPhases(sliceSize); + std::vector prevMask(sliceSize); + std::vector curMask(sliceSize); + std::vector nextMask(sliceSize); + + // Helper to load a mask slice from the store + auto loadMaskSlice = [&](usize offset, std::vector& dest) { + if(maskStorePtr != nullptr) + { + maskStorePtr->copyIntoBuffer(offset, nonstd::span(dest.data(), sliceSize)); + } + else + { + // Bool mask: read per-element via maskCompare + for(usize i = 0; i < sliceSize; i++) + { + dest[i] = maskCompare->isTrue(offset + i) ? 1 : 0; + } + } + }; + + // Helper to load all 3 arrays for a given Z-slice into dest buffers + auto loadSlice = [&](int64 z, std::vector& dstQuats, std::vector& dstPhases, std::vector& dstMask) { + const usize offset = static_cast(z) * sliceSize; + quatsStore.copyIntoBuffer(offset * 4, nonstd::span(dstQuats.data(), quatSliceElems)); + phasesStore.copyIntoBuffer(offset, nonstd::span(dstPhases.data(), sliceSize)); + loadMaskSlice(offset, dstMask); + }; + + // Multi-level iterative flipping with on-the-fly neighbor count recomputation. + // No precomputed neighborCount array — counts are recomputed per voxel per pass. + constexpr int32 startLevel = 6; + const int32 totalLevels = startLevel - m_InputValues->NumberOfNeighbors + 1; + + for(int32 currentLevel = startLevel; currentLevel >= m_InputValues->NumberOfNeighbors; currentLevel--) + { + bool changed = true; + int32 passCount = 0; + + while(changed) + { + changed = false; + passCount++; + + // Load initial slices for this pass + loadSlice(0, curQuats, curPhases, curMask); + if(dimZ > 1) + { + loadSlice(1, nextQuats, nextPhases, nextMask); + } + + for(int64 zIdx = 0; zIdx < dimZ; zIdx++) + { + if(m_ShouldCancel) + { + return {}; + } + + bool sliceChanged = false; + for(int64 yIdx = 0; yIdx < dimY; yIdx++) + { + const int64 jStride = yIdx * dimX; + for(int64 xIdx = 0; xIdx < dimX; xIdx++) + { + const int64 sliceIndex = jStride + xIdx; + + if(curMask[sliceIndex]) + { + continue; // already good + } + + const int64 quatOffset = sliceIndex * 4; + ebsdlib::QuatD quat1(curQuats[quatOffset], curQuats[quatOffset + 1], curQuats[quatOffset + 2], curQuats[quatOffset + 3]); + quat1.positiveOrientation(); + const int32 curPhase = curPhases[sliceIndex]; + const uint32 laueClass = localCrystalStructures[curPhase]; + + int32 count = countMatchingNeighbors(xIdx, yIdx, zIdx, dimX, dimY, dimZ, sliceIndex, prevQuats, curQuats, nextQuats, prevPhases, curPhases, nextPhases, prevMask, curMask, nextMask, + curPhase, laueClass, quat1, misorientationTolerance, orientationOps); + + if(count >= currentLevel) + { + // Flip this voxel in the local mask buffer (takes effect for + // subsequent voxels in this slice and as prevMask for the next slice) + curMask[sliceIndex] = 1; + sliceChanged = true; + } + } + } + + // Write back any mask changes for this Z-slice to the real store + if(sliceChanged) + { + changed = true; + const usize sliceOffset = static_cast(zIdx) * sliceSize; + if(maskStorePtr != nullptr) + { + maskStorePtr->copyFromBuffer(sliceOffset, nonstd::span(curMask.data(), sliceSize)); + } + else + { + for(usize i = 0; i < sliceSize; i++) + { + maskCompare->setValue(sliceOffset + i, curMask[i] != 0); + } + } + } + + // Shift rolling window + std::swap(prevQuats, curQuats); + std::swap(curQuats, nextQuats); + std::swap(prevPhases, curPhases); + std::swap(curPhases, nextPhases); + std::swap(prevMask, curMask); + std::swap(curMask, nextMask); + if(zIdx + 2 < dimZ) + { + loadSlice(zIdx + 2, nextQuats, nextPhases, nextMask); + } + } + } + } + + return {}; +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp new file mode 100644 index 0000000000..1057ab1ab4 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp @@ -0,0 +1,35 @@ +#pragma once + +#include "OrientationAnalysis/OrientationAnalysis_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct BadDataNeighborOrientationCheckInputValues; + +class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckScanline +{ +public: + BadDataNeighborOrientationCheckScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const BadDataNeighborOrientationCheckInputValues* inputValues); + ~BadDataNeighborOrientationCheckScanline() noexcept; + + BadDataNeighborOrientationCheckScanline(const BadDataNeighborOrientationCheckScanline&) = delete; + BadDataNeighborOrientationCheckScanline(BadDataNeighborOrientationCheckScanline&&) noexcept = delete; + BadDataNeighborOrientationCheckScanline& operator=(const BadDataNeighborOrientationCheckScanline&) = delete; + BadDataNeighborOrientationCheckScanline& operator=(BadDataNeighborOrientationCheckScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp index 1c862c6a5e..c82e84904b 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp @@ -1,19 +1,22 @@ #include "BadDataNeighborOrientationCheckWorklist.hpp" +#include "BadDataNeighborOrientationCheck.hpp" + #include "simplnx/Common/Numbers.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/MaskCompareUtilities.hpp" -#include "simplnx/Utilities/MessageHelper.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" #include +#include + using namespace nx::core; // ----------------------------------------------------------------------------- BadDataNeighborOrientationCheckWorklist::BadDataNeighborOrientationCheckWorklist(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - BadDataNeighborOrientationCheckInputValues* inputValues) + const BadDataNeighborOrientationCheckInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -25,20 +28,20 @@ BadDataNeighborOrientationCheckWorklist::BadDataNeighborOrientationCheckWorklist BadDataNeighborOrientationCheckWorklist::~BadDataNeighborOrientationCheckWorklist() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& BadDataNeighborOrientationCheckWorklist::getCancel() -{ - return m_ShouldCancel; -} - -// ----------------------------------------------------------------------------- +/** + * @brief Flips bad voxels to good using a worklist-based propagation algorithm. + * In-core path: Phase 1 counts matching good neighbors per bad voxel. + * Phase 2 iteratively flips eligible voxels using a deque worklist, + * propagating new eligibility to neighbors immediately. + */ Result<> BadDataNeighborOrientationCheckWorklist::operator()() { - const float misorientationTolerance = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; + const float32 misorientationTolerance = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; const auto* imageGeomPtr = m_DataStructure.getDataAs(m_InputValues->ImageGeomPath); SizeVec3 udims = imageGeomPtr->getDimensions(); const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); - const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); + auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); const usize totalPoints = quats.getNumberOfTuples(); @@ -48,8 +51,6 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); } catch(const std::out_of_range& exception) { - // This really should NOT be happening as the path was verified during preflight, BUT we may be calling this from - // somewhere else that is NOT going through the normal nx::core::IFilter API of Preflight and Execute return MakeErrorResult(-54900, fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString())); } @@ -59,6 +60,8 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() static_cast(udims[2]), }; + const int64 xyStride = dims[0] * dims[1]; + std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); @@ -66,26 +69,19 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() std::vector neighborCount(totalPoints, 0); - MessageHelper messageHelper(m_MessageHandler); - ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); - // Loop over every point finding the number of neighbors that fall within the - // user defined angle tolerance. - for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) + // ===== Phase 1: Count matching good neighbors for each bad voxel ===== + for(usize voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) { - throttledMessenger.sendThrottledMessage([&] { return fmt::format("Processing Data {:.2f}% completed", CalculatePercentComplete(voxelIndex, totalPoints)); }); - // If the mask was set to false, then we check this voxel if(!maskCompare->isTrue(voxelIndex)) { - // We precalculate the positive voxel quaternion and laue class here to prevent reading and recalculating it for each face below ebsdlib::QuatD quat1(quats[voxelIndex * 4], quats[voxelIndex * 4 + 1], quats[voxelIndex * 4 + 2], quats[voxelIndex * 4 + 3]); quat1.positiveOrientation(); const uint32 laueClass1 = crystalStructures[cellPhases[voxelIndex]]; - int64 xIdx = voxelIndex % dims[0]; - int64 yIdx = (voxelIndex / dims[0]) % dims[1]; - int64 zIdx = voxelIndex / (dims[0] * dims[1]); + const int64 xIdx = static_cast(voxelIndex) % dims[0]; + const int64 yIdx = (static_cast(voxelIndex) / dims[0]) % dims[1]; + const int64 zIdx = static_cast(voxelIndex) / xyStride; - // Loop over the 6 face neighbors of the voxel std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); for(const auto& faceIndex : faceNeighborInternalIdx) { @@ -93,22 +89,15 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() { continue; } - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + const int64 neighborPoint = static_cast(voxelIndex) + neighborVoxelIndexOffsets[faceIndex]; - // Now compare the mask of the neighbor. If the mask is TRUE, i.e., that voxel - // did not fail the threshold filter that most likely produced the mask array, - // then we can look at that voxel. if(maskCompare->isTrue(neighborPoint)) { - // Both Cell Phases MUST be the same and be a valid Phase if(cellPhases[voxelIndex] == cellPhases[neighborPoint] && cellPhases[voxelIndex] > 0) { ebsdlib::QuatD quat2(quats[neighborPoint * 4], quats[neighborPoint * 4 + 1], quats[neighborPoint * 4 + 2], quats[neighborPoint * 4 + 3]); quat2.positiveOrientation(); - // Compute the Axis_Angle misorientation between those 2 quaternions ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); - // if the angle is less than our tolerance, then we increment the neighbor count - // for this voxel if(axisAngle[3] < misorientationTolerance) { neighborCount[voxelIndex]++; @@ -119,83 +108,70 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() } } + // ===== Phase 2: Iteratively flip bad voxels using worklist ===== constexpr int32 startLevel = 6; - int32 currentLevel = startLevel; - int32 counter = 0; + const int32 totalLevels = startLevel - m_InputValues->NumberOfNeighbors + 1; - // Now we loop over all the points again, but this time we do it as many times - // as the user has requested to iteratively flip voxels - while(currentLevel >= m_InputValues->NumberOfNeighbors) + for(int32 currentLevel = startLevel; currentLevel >= m_InputValues->NumberOfNeighbors; currentLevel--) { - counter = 1; - int32 loopNumber = 0; - while(counter > 0) + std::deque worklist; + for(usize voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) { - counter = 0; // Set this while control variable to zero - for(usize voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) + if(neighborCount[voxelIndex] >= currentLevel && !maskCompare->isTrue(voxelIndex)) { - throttledMessenger.sendThrottledMessage([&] { - return fmt::format("Level '{}' of '{}' || Processing Data ('{}') {:.2f}% completed", (startLevel - currentLevel) + 1, startLevel - m_InputValues->NumberOfNeighbors, loopNumber, - CalculatePercentComplete(voxelIndex, totalPoints)); - }); - - // We are comparing the number-of-neighbors of the current voxel, and if it - // is > the current level and the mask is FALSE, then we drop into this - // conditional. The first thing that happens in the conditional is that - // the current voxel's mask value is set to TRUE. - if(neighborCount[voxelIndex] >= currentLevel && !maskCompare->isTrue(voxelIndex)) + worklist.push_back(voxelIndex); + } + } + + while(!worklist.empty()) + { + const usize voxelIndex = worklist.front(); + worklist.pop_front(); + + if(maskCompare->isTrue(voxelIndex) || neighborCount[voxelIndex] < currentLevel) + { + continue; + } + + maskCompare->setValue(voxelIndex, true); + + ebsdlib::QuatD quat1(quats[voxelIndex * 4], quats[voxelIndex * 4 + 1], quats[voxelIndex * 4 + 2], quats[voxelIndex * 4 + 3]); + quat1.positiveOrientation(); + const uint32 laueClass1 = crystalStructures[cellPhases[voxelIndex]]; + + const int64 xIdx = static_cast(voxelIndex) % dims[0]; + const int64 yIdx = (static_cast(voxelIndex) / dims[0]) % dims[1]; + const int64 zIdx = static_cast(voxelIndex) / xyStride; + + std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + for(const auto& faceIndex : faceNeighborInternalIdx) + { + if(!isValidFaceNeighbor[faceIndex]) { - maskCompare->setValue(voxelIndex, true); // the current voxel's mask value is set to TRUE. - counter++; // Increment the `counter` to force the loop to iterate again - - // We precalculate the positive voxel quaternion and laue class here to prevent reading and recalculating it for each face below - ebsdlib::QuatD quat1(quats[voxelIndex * 4], quats[voxelIndex * 4 + 1], quats[voxelIndex * 4 + 2], quats[voxelIndex * 4 + 3]); - quat1.positiveOrientation(); - const uint32 laueClass1 = crystalStructures[cellPhases[voxelIndex]]; - - // This whole section below is to now look at the neighbor voxels of the - // current voxel that just got flipped to true. This is needed because - // if any of those neighbor's mask was `false`, then its neighbor count - // is now not correct and will be off-by-one. So we run _almost_ the same - // loop code as above but checking the specific neighbors of the current - // voxel. This part should be termed the "Update Neighbor's Neighbor Count" - int64 xIdx = voxelIndex % dims[0]; - int64 yIdx = (voxelIndex / dims[0]) % dims[1]; - int64 zIdx = voxelIndex / (dims[0] * dims[1]); - - // Loop over the 6 face neighbors of the voxel - std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(const auto& faceIndex : faceNeighborInternalIdx) - { - if(!isValidFaceNeighbor[faceIndex]) - { - continue; - } + continue; + } - int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + const int64 neighborPoint = static_cast(voxelIndex) + neighborVoxelIndexOffsets[faceIndex]; - // If the neighbor voxel's mask is false, then .... - if(!maskCompare->isTrue(neighborPoint)) + if(!maskCompare->isTrue(neighborPoint)) + { + if(cellPhases[voxelIndex] == cellPhases[neighborPoint] && cellPhases[voxelIndex] > 0) + { + ebsdlib::QuatD quat2(quats[neighborPoint * 4], quats[neighborPoint * 4 + 1], quats[neighborPoint * 4 + 2], quats[neighborPoint * 4 + 3]); + quat2.positiveOrientation(); + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); + if(axisAngle[3] < misorientationTolerance) { - // Make sure both cells phase values are identical and valid - if(cellPhases[voxelIndex] == cellPhases[neighborPoint] && cellPhases[voxelIndex] > 0) + neighborCount[neighborPoint]++; + if(neighborCount[neighborPoint] >= currentLevel) { - ebsdlib::QuatD quat2(quats[neighborPoint * 4], quats[neighborPoint * 4 + 1], quats[neighborPoint * 4 + 2], quats[neighborPoint * 4 + 3]); - quat2.positiveOrientation(); - // Quaternion Math is not commutative so do not reorder - ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); - if(axisAngle[3] < misorientationTolerance) - { - neighborCount[neighborPoint]++; - } + worklist.push_back(static_cast(neighborPoint)); } } } } } - ++loopNumber; } - currentLevel = currentLevel - 1; } return {}; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp index 1a2cd42b4e..5bbba4e92c 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp @@ -9,25 +9,13 @@ namespace nx::core { -struct ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckInputValues -{ - float32 MisorientationTolerance; - int32 NumberOfNeighbors; - DataPath ImageGeomPath; - DataPath QuatsArrayPath; - DataPath MaskArrayPath; - DataPath CellPhasesArrayPath; - DataPath CrystalStructuresArrayPath; -}; +struct BadDataNeighborOrientationCheckInputValues; -/** - * @class - */ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckWorklist { public: BadDataNeighborOrientationCheckWorklist(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - BadDataNeighborOrientationCheckInputValues* inputValues); + const BadDataNeighborOrientationCheckInputValues* inputValues); ~BadDataNeighborOrientationCheckWorklist() noexcept; BadDataNeighborOrientationCheckWorklist(const BadDataNeighborOrientationCheckWorklist&) = delete; @@ -37,8 +25,6 @@ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckWorklist Result<> operator()(); - const std::atomic_bool& getCancel(); - private: DataStructure& m_DataStructure; const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.cpp index 587ecbf6c3..67c4e1c1dd 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.cpp @@ -5,13 +5,16 @@ #include "simplnx/Common/Constants.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/ClusteringUtilities.hpp" #include #include #include +#include #include +#include using namespace nx::core; using namespace nx::core::OrientationUtilities; @@ -26,6 +29,37 @@ CAxisSegmentFeatures::CAxisSegmentFeatures(DataStructure& dataStructure, const I // ----------------------------------------------------------------------------- CAxisSegmentFeatures::~CAxisSegmentFeatures() noexcept = default; +// ----------------------------------------------------------------------------- +// Segments a hexagonal EBSD dataset into features (grains) based on c-axis +// alignment. Two neighboring voxels are grouped into the same feature when +// their crystallographic c-axes (the [0001] direction) are aligned within a +// user-specified angular tolerance. Unlike EBSDSegmentFeatures which uses full +// misorientation via LaueOps, this filter only considers the c-axis direction, +// which is useful for analyzing basal texture in hexagonal materials. +// +// Pre-validation: +// Before segmentation, every cell's phase is checked against the crystal +// structure table. All phases must be hexagonal (Hexagonal_High 6/mmm or +// Hexagonal_Low 6/m); if any non-hexagonal phase is found, the filter +// returns an error because c-axis alignment is only meaningful for HCP. +// +// Algorithm dispatch: +// - In-core data -> execute() : classic depth-first-search (DFS) flood fill +// - Out-of-core -> executeCCL() : connected-component labeling that streams +// data slice-by-slice to limit memory usage +// The choice is made by checking IsOutOfCore() on the FeatureIds array (i.e., +// whether the backing DataStore lives on disk) or if ForceOocAlgorithm() is +// set (used for testing). +// +// Post-processing after either algorithm: +// 1. Validate that at least one feature was found (error if not). +// 2. Resize the Feature AttributeMatrix to (m_FoundFeatures + 1) tuples so +// that all per-feature arrays (Active, etc.) have the correct size. +// Index 0 is reserved as an invalid/background feature. +// 3. Initialize the Active array: fill with 1 (active), then set index 0 +// to 0 to mark it as the reserved background slot. +// 4. Optionally randomize FeatureIds so that spatially adjacent grains get +// non-sequential IDs, improving visual contrast in color-mapped renders. // ----------------------------------------------------------------------------- Result<> CAxisSegmentFeatures::operator()() { @@ -49,17 +83,34 @@ Result<> CAxisSegmentFeatures::operator()() // Loop through all the "Phase" cell values and validate that any phase found is // a hexagonal phase. This guards against there being multiple phases defined in // and EBSD file but the non-hexagonal phases were actually never found - const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + + // Cache crystal structures locally to avoid per-element OOC access (principle 9) + const usize numPhases = crystalStructures.getNumberOfTuples(); + std::vector crystalStructuresCache(numPhases); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.data(), numPhases)); + usize numCells = m_CellPhases->getNumberOfTuples(); - for(usize cellIdx = 0; cellIdx < numCells; ++cellIdx) + // Use a Z-slice-sized batch for optimal OOC I/O (one HDF5 hyperslab per slice) + SizeVec3 scanDims = imageGeometry->getDimensions(); + const usize k_ScanBatchSize = static_cast(scanDims[0]) * static_cast(scanDims[1]); + auto phasesBuf = std::make_unique(k_ScanBatchSize); + auto& phasesStore = m_CellPhases->getDataStoreRef(); + for(usize offset = 0; offset < numCells; offset += k_ScanBatchSize) { - int32 currentPhaseIdx = m_CellPhases->getValue(cellIdx); - const auto crystalStructureType = crystalStructures[currentPhaseIdx]; - if(crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_High && crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_Low) + const usize batchSize = std::min(k_ScanBatchSize, numCells - offset); + phasesStore.copyIntoBuffer(offset, nonstd::span(phasesBuf.get(), batchSize)); + for(usize i = 0; i < batchSize; i++) { - return MakeErrorResult(-8363, fmt::format("Input data is using {} type crystal structures but segmenting features via c-axis mis orientation requires all phases to be either Hexagonal-Low 6/m " - "or Hexagonal-High 6/mmm type crystal structures.", - CrystalStructureEnumToString(crystalStructureType))); + int32 currentPhaseIdx = phasesBuf[i]; + const auto crystalStructureType = crystalStructuresCache[static_cast(currentPhaseIdx)]; + if(crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_High && crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_Low) + { + return MakeErrorResult(-8363, + fmt::format("Input data is using {} type crystal structures but segmenting features via c-axis mis orientation requires all phases to be either Hexagonal-Low 6/m " + "or Hexagonal-High 6/mmm type crystal structures.", + CrystalStructureEnumToString(crystalStructureType))); + } } } @@ -68,8 +119,21 @@ Result<> CAxisSegmentFeatures::operator()() auto* active = m_DataStructure.getDataAs(m_InputValues->ActiveArrayPath); active->fill(1); - // Run the segmentation algorithm - execute(imageGeometry); + // Dispatch between DFS (in-core) and CCL (OOC) algorithms + if(IsOutOfCore(*m_FeatureIdsArray) || ForceOocAlgorithm()) + { + SizeVec3 udims = imageGeometry->getDimensions(); + allocateSliceBuffers(static_cast(udims[0]), static_cast(udims[1])); + + auto& featureIdsStore = m_FeatureIdsArray->getDataStoreRef(); + executeCCL(imageGeometry, featureIdsStore); + + deallocateSliceBuffers(); + } + else + { + execute(imageGeometry); + } // Sanity check the result. if(this->m_FoundFeatures < 1) { @@ -97,6 +161,22 @@ Result<> CAxisSegmentFeatures::operator()() return {}; } +// ----------------------------------------------------------------------------- +// Finds the next unassigned voxel that can serve as the seed for a new feature. +// The scan is a simple linear walk starting from `nextSeed`, which is the index +// immediately after the last seed found. This avoids rescanning already-assigned +// voxels at the beginning of the array. +// +// A voxel is eligible to become a seed when all three conditions are met: +// 1. featureId == 0 : the voxel has not yet been assigned to any feature. +// 2. Passes the mask: if masking is enabled, the voxel must be flagged as +// "good" (e.g., not a bad scan point). +// 3. cellPhase > 0 : the voxel belongs to a real crystallographic phase +// (phase 0 is reserved for unindexed/background points). +// +// When a valid seed is found, its featureId is immediately set to `gnum` +// (the new feature number) so that subsequent calls will skip it. +// Returns the linear index of the seed, or -1 if no more seeds exist. // ----------------------------------------------------------------------------- int64 CAxisSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const { @@ -127,14 +207,34 @@ int64 CAxisSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const } if(seed >= 0) { - auto& cellFeatureAM = m_DataStructure.getDataRefAs(m_InputValues->CellFeatureAttributeMatrixPath); featureIds[static_cast(seed)] = gnum; - const ShapeType tDims = {static_cast(gnum) + 1}; - cellFeatureAM.resizeTuples(tDims); // This will resize the active array } return seed; } +// ----------------------------------------------------------------------------- +// Determines whether a neighboring voxel should be merged into the current +// feature during the DFS flood fill (execute() path). This is NOT used by +// the CCL path, which calls areNeighborsSimilar() instead. +// +// The method checks three conditions before grouping: +// 1. The neighbor's featureId must be 0 (unassigned). +// 2. The neighbor must pass the mask (if masking is enabled). +// 3. The neighbor must have a c-axis aligned with the reference voxel. +// +// C-axis misalignment calculation: +// - Both voxels must share the same phase (no cross-phase grouping). +// - Quaternion orientations (QuatF, 4 floats) are extracted for both voxels. +// - Each quaternion is converted to a 3x3 orientation matrix, which is then +// transposed and multiplied by the crystal c-axis unit vector [0,0,1] to +// obtain the sample-frame c-axis direction for each voxel. +// - Both c-axis vectors are normalized so the dot product directly gives +// the cosine of the angle between them. +// - The dot product is clamped to [-1, 1] to guard against floating-point +// error, then acos() gives the misalignment angle w (in radians). +// - Because the c-axis is bidirectional (parallel and antiparallel are +// equivalent), the check accepts w <= tolerance OR (pi - w) <= tolerance. +// - If accepted, the neighbor's featureId is set to `gnum` as a side effect. // ----------------------------------------------------------------------------- bool CAxisSegmentFeatures::determineGrouping(int64 referencepoint, int64 neighborpoint, int32 gnum) const { @@ -182,3 +282,309 @@ bool CAxisSegmentFeatures::determineGrouping(int64 referencepoint, int64 neighbo } return group; } + +// ----------------------------------------------------------------------------- +// Checks whether a single voxel is eligible for segmentation (used by the CCL +// path in executeCCL()). A voxel is valid if it passes the mask and has a +// crystallographic phase > 0. +// +// Slice buffer fast path: +// When m_UseSliceBuffers is true (OOC mode), the method checks whether the +// voxel's Z-slice is currently loaded in one of the two buffer slots. The +// slot lookup checks both m_BufferedSliceZ[0] and m_BufferedSliceZ[1] to +// find which slot (if any) holds the target slice. If found, mask and phase +// values are read from the in-memory m_MaskBuffer and m_PhaseBuffer arrays, +// avoiding an on-disk I/O round-trip. +// +// OOC fallback: +// If slice buffers are not active, or if the voxel's slice is not currently +// buffered (which can happen during Phase 1b of CCL when periodic boundary +// merging accesses non-adjacent slices), the method falls back to direct +// array access through the DataStore, which may trigger on-disk I/O for +// out-of-core data. +// ----------------------------------------------------------------------------- +bool CAxisSegmentFeatures::isValidVoxel(int64 point) const +{ + if(m_UseSliceBuffers) + { + int64 sliceZ = point / m_BufSliceSize; + if(sliceZ == m_BufferedSliceZ[0] || sliceZ == m_BufferedSliceZ[1]) + { + int64 slot = (sliceZ == m_BufferedSliceZ[0]) ? 0 : 1; + int64 offset = point - sliceZ * m_BufSliceSize; + int64 bufIdx = slot * m_BufSliceSize + offset; + // Check mask + if(m_InputValues->UseMask && m_MaskBuffer[bufIdx] == 0) + { + return false; + } + // Check phase + if(m_PhaseBuffer[bufIdx] <= 0) + { + return false; + } + return true; + } + } + + // Fallback: direct array access + if(m_InputValues->UseMask && !m_GoodVoxelsArray->isTrue(point)) + { + return false; + } + Int32Array& cellPhases = *m_CellPhases; + if(cellPhases[point] <= 0) + { + return false; + } + return true; +} + +// ----------------------------------------------------------------------------- +// Determines whether two neighboring voxels have sufficiently aligned c-axes +// to belong to the same feature. Used exclusively by the CCL path +// (executeCCL()), whereas the DFS path uses determineGrouping() instead. +// +// Slice buffer fast path: +// When both voxels' Z-slices are present in the rolling 2-slot buffer, all +// data is read from the in-memory buffers (m_QuatBuffer, m_PhaseBuffer, +// m_MaskBuffer). The buffer index for each point is computed as: +// slot * sliceSize + (point - sliceZ * sliceSize) +// For quaternions, an additional x4 factor accounts for the 4 components +// per voxel. The method then: +// 1. Checks point2's mask validity. +// 2. Checks that point2's phase > 0 and both phases match. +// 3. Constructs QuatF objects from the buffered quaternion components. +// 4. Converts each quaternion to an orientation matrix, transposes it, and +// multiplies by [0,0,1] to get the sample-frame c-axis direction. +// 5. Normalizes both c-axis vectors and computes the dot product. +// 6. Clamps the dot product to [-1,1] and takes acos() to get the +// misalignment angle w. +// 7. Returns true if w <= tolerance OR (pi - w) <= tolerance (because +// parallel and antiparallel c-axes are crystallographically equivalent). +// +// OOC fallback: +// If either voxel's slice is not buffered (e.g., during Phase 1b periodic +// merge), falls back to direct DataStore access: validates point2 via +// isValidVoxel(), checks phase equality, then computes c-axis misalignment +// from the full quaternion and phase arrays on disk. +// ----------------------------------------------------------------------------- +bool CAxisSegmentFeatures::areNeighborsSimilar(int64 point1, int64 point2) const +{ + if(m_UseSliceBuffers) + { + int64 sliceZ1 = point1 / m_BufSliceSize; + int64 sliceZ2 = point2 / m_BufSliceSize; + bool buf1 = (sliceZ1 == m_BufferedSliceZ[0] || sliceZ1 == m_BufferedSliceZ[1]); + bool buf2 = (sliceZ2 == m_BufferedSliceZ[0] || sliceZ2 == m_BufferedSliceZ[1]); + + if(buf1 && buf2) + { + int64 slot1 = (sliceZ1 == m_BufferedSliceZ[0]) ? 0 : 1; + int64 slot2 = (sliceZ2 == m_BufferedSliceZ[0]) ? 0 : 1; + int64 off1 = point1 - sliceZ1 * m_BufSliceSize; + int64 off2 = point2 - sliceZ2 * m_BufSliceSize; + int64 bufIdx1 = slot1 * m_BufSliceSize + off1; + int64 bufIdx2 = slot2 * m_BufSliceSize + off2; + + // Check point2 validity (mask + phase) + if(m_InputValues->UseMask && m_MaskBuffer[bufIdx2] == 0) + { + return false; + } + if(m_PhaseBuffer[bufIdx2] <= 0) + { + return false; + } + + // Must be same phase + if(m_PhaseBuffer[bufIdx1] != m_PhaseBuffer[bufIdx2]) + { + return false; + } + + // Read quaternions from buffer + int64 qIdx1 = bufIdx1 * 4; + int64 qIdx2 = bufIdx2 * 4; + const ebsdlib::QuatF q1(m_QuatBuffer[qIdx1], m_QuatBuffer[qIdx1 + 1], m_QuatBuffer[qIdx1 + 2], m_QuatBuffer[qIdx1 + 3]); + const ebsdlib::QuatF q2(m_QuatBuffer[qIdx2], m_QuatBuffer[qIdx2 + 1], m_QuatBuffer[qIdx2 + 2], m_QuatBuffer[qIdx2 + 3]); + + const ebsdlib::OrientationMatrixFType oMatrix1 = q1.toOrientationMatrix(); + const ebsdlib::OrientationMatrixFType oMatrix2 = q2.toOrientationMatrix(); + + const Eigen::Vector3f cAxis{0.0f, 0.0f, 1.0f}; + Eigen::Vector3f c1 = oMatrix1.transpose() * cAxis; + Eigen::Vector3f c2 = oMatrix2.transpose() * cAxis; + + c1.normalize(); + c2.normalize(); + + float32 w = std::clamp(((c1[0] * c2[0]) + (c1[1] * c2[1]) + (c1[2] * c2[2])), -1.0F, 1.0F); + w = std::acos(w); + + return w <= m_InputValues->MisorientationTolerance || (Constants::k_PiD - w) <= m_InputValues->MisorientationTolerance; + } + } + + // Fallback: direct array access + if(!isValidVoxel(point2)) + { + return false; + } + + Int32Array& cellPhases = *m_CellPhases; + + // Must be same phase + if(cellPhases[point1] != cellPhases[point2]) + { + return false; + } + + // Calculate c-axis misalignment + const Eigen::Vector3f cAxis{0.0f, 0.0f, 1.0f}; + Float32Array& quats = *m_QuatsArray; + + const ebsdlib::QuatF q1(quats[point1 * 4], quats[point1 * 4 + 1], quats[point1 * 4 + 2], quats[point1 * 4 + 3]); + const ebsdlib::QuatF q2(quats[point2 * 4], quats[point2 * 4 + 1], quats[point2 * 4 + 2], quats[point2 * 4 + 3]); + + const ebsdlib::OrientationMatrixFType oMatrix1 = q1.toOrientationMatrix(); + const ebsdlib::OrientationMatrixFType oMatrix2 = q2.toOrientationMatrix(); + + Eigen::Vector3f c1 = oMatrix1.transpose() * cAxis; + Eigen::Vector3f c2 = oMatrix2.transpose() * cAxis; + + c1.normalize(); + c2.normalize(); + + float32 w = std::clamp(((c1[0] * c2[0]) + (c1[1] * c2[1]) + (c1[2] * c2[2])), -1.0F, 1.0F); + w = std::acos(w); + + return w <= m_InputValues->MisorientationTolerance || (Constants::k_PiD - w) <= m_InputValues->MisorientationTolerance; +} + +// ----------------------------------------------------------------------------- +// Allocates the rolling 2-slot slice buffers used by the CCL (OOC) algorithm. +// Called once at the start of the OOC branch in operator(), before executeCCL(). +// +// Each slot holds one full XY slice (dimX * dimY voxels). Two slots are needed +// because the CCL algorithm compares the current slice (iz) with the previous +// slice (iz-1), so both must be in memory simultaneously. +// +// Buffers allocated: +// - m_QuatBuffer : 2 * sliceSize * 4 floats (quaternion: 4 components/voxel) +// - m_PhaseBuffer : 2 * sliceSize int32 values (one phase ID per voxel) +// - m_MaskBuffer : 2 * sliceSize uint8 values (one mask flag per voxel) +// +// Both m_BufferedSliceZ slots are initialized to -1 (no slice loaded). +// m_UseSliceBuffers is set to true so that isValidVoxel() and +// areNeighborsSimilar() will use the fast buffer path. +// ----------------------------------------------------------------------------- +void CAxisSegmentFeatures::allocateSliceBuffers(int64 dimX, int64 dimY) +{ + m_BufSliceSize = dimX * dimY; + int64 totalSlots = 2 * m_BufSliceSize; + m_QuatBuffer.resize(static_cast(totalSlots * 4)); + m_PhaseBuffer.resize(static_cast(totalSlots)); + m_MaskBuffer.resize(static_cast(totalSlots)); + m_BufferedSliceZ[0] = -1; + m_BufferedSliceZ[1] = -1; + m_UseSliceBuffers = true; +} + +// ----------------------------------------------------------------------------- +// Releases the slice buffers after executeCCL() completes, freeing the memory +// back to the system. Called in the OOC branch of operator() after the CCL +// algorithm finishes. Resets m_UseSliceBuffers to false and both +// m_BufferedSliceZ slots to -1. Uses clear() + shrink_to_fit() on each vector +// to guarantee memory deallocation. +// ----------------------------------------------------------------------------- +void CAxisSegmentFeatures::deallocateSliceBuffers() +{ + m_UseSliceBuffers = false; + m_QuatBuffer.clear(); + m_QuatBuffer.shrink_to_fit(); + m_PhaseBuffer.clear(); + m_PhaseBuffer.shrink_to_fit(); + m_MaskBuffer.clear(); + m_MaskBuffer.shrink_to_fit(); + m_BufferedSliceZ[0] = -1; + m_BufferedSliceZ[1] = -1; + m_BufSliceSize = 0; +} + +// ----------------------------------------------------------------------------- +// Pre-loads voxel data for a single Z-slice into the rolling 2-slot buffer, +// called by executeCCL() before processing each slice. +// +// Rolling buffer design: +// The target slot is determined by (iz % 2), so even slices go to slot 0 and +// odd slices go to slot 1. Because the CCL algorithm processes slices in +// order (0, 1, 2, ...), at any given slice iz the previous slice (iz-1) is +// always in the other slot, keeping both the current and previous slice data +// available in memory. +// +// Sentinel behavior: +// If iz < 0, slice buffering is disabled (m_UseSliceBuffers = false). The +// CCL algorithm passes iz = -1 after completing the slice-by-slice sweep to +// signal that subsequent calls (e.g., during Phase 1b periodic boundary +// merging) should use direct DataStore access instead of the buffers. +// +// Data loaded per slice: +// - Quaternions (4 float32 per voxel) into m_QuatBuffer +// - Phase IDs (1 int32 per voxel) into m_PhaseBuffer +// - Mask flags (1 uint8 per voxel) into m_MaskBuffer; if masking is disabled, +// all mask values are set to 1 (valid) +// +// Note: Unlike the EBSDSegmentFeatures version, this implementation does not +// include a skip-if-already-loaded check; the slot is always overwritten. +// ----------------------------------------------------------------------------- +void CAxisSegmentFeatures::prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) +{ + if(iz < 0) + { + m_UseSliceBuffers = false; + return; + } + + int64 slot = iz % 2; + m_BufferedSliceZ[slot] = iz; + + const int64 sliceStart = iz * m_BufSliceSize; + const int64 bufOffset = slot * m_BufSliceSize; + const usize sliceSize = static_cast(m_BufSliceSize); + const usize slotOffset = static_cast(bufOffset); + + // Bulk-read quaternions (4 components per voxel) for this slice + AbstractDataStore& quatStore = m_QuatsArray->getDataStoreRef(); + quatStore.copyIntoBuffer(static_cast(sliceStart) * 4, nonstd::span(m_QuatBuffer.data() + slotOffset * 4, sliceSize * 4)); + + // Bulk-read phase IDs for this slice + AbstractDataStore& phaseStore = m_CellPhases->getDataStoreRef(); + phaseStore.copyIntoBuffer(static_cast(sliceStart), nonstd::span(m_PhaseBuffer.data() + slotOffset, sliceSize)); + + // Bulk-read mask flags for this slice + if(m_InputValues->UseMask && m_GoodVoxelsArray != nullptr) + { + auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + if(maskArray.getDataType() == DataType::uint8) + { + auto& typedStore = maskArray.getIDataStoreRefAs>(); + typedStore.copyIntoBuffer(static_cast(sliceStart), nonstd::span(m_MaskBuffer.data() + slotOffset, sliceSize)); + } + else if(maskArray.getDataType() == DataType::boolean) + { + auto& typedStore = maskArray.getIDataStoreRefAs>(); + auto boolBuf = std::make_unique(sliceSize); + typedStore.copyIntoBuffer(static_cast(sliceStart), nonstd::span(boolBuf.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + m_MaskBuffer[slotOffset + i] = boolBuf[i] ? 1 : 0; + } + } + } + else + { + // If no mask, mark everything as valid + std::fill(m_MaskBuffer.begin() + slotOffset, m_MaskBuffer.begin() + slotOffset + sliceSize, static_cast(1)); + } +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp index b1d0fe9d88..343a16301e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp @@ -49,6 +49,23 @@ class ORIENTATIONANALYSIS_EXPORT CAxisSegmentFeatures : public SegmentFeatures int64 getSeed(int32 gnum, int64 nextSeed) const override; bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; + /** + * @brief Checks whether a voxel can participate in C-axis segmentation based on mask and phase. + * @param point Linear voxel index. + * @return true if the voxel passes mask and phase checks. + */ + bool isValidVoxel(int64 point) const override; + + /** + * @brief Determines whether two neighboring voxels belong to the same C-axis segment. + * @param point1 First voxel index. + * @param point2 Second (neighbor) voxel index. + * @return true if both voxels share the same phase and their C-axis misalignment is within tolerance. + */ + bool areNeighborsSimilar(int64 point1, int64 point2) const override; + + void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) override; + private: const CAxisSegmentFeaturesInputValues* m_InputValues = nullptr; @@ -56,6 +73,17 @@ class ORIENTATIONANALYSIS_EXPORT CAxisSegmentFeatures : public SegmentFeatures Int32Array* m_CellPhases = nullptr; std::unique_ptr m_GoodVoxelsArray = nullptr; Int32Array* m_FeatureIdsArray = nullptr; + + void allocateSliceBuffers(int64 dimX, int64 dimY); + void deallocateSliceBuffers(); + + // Rolling 2-slot input buffers for OOC optimization. + std::vector m_QuatBuffer; + std::vector m_PhaseBuffer; + std::vector m_MaskBuffer; + int64 m_BufSliceSize = 0; + int64 m_BufferedSliceZ[2] = {-1, -1}; + bool m_UseSliceBuffers = false; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp index 04fe595c5b..bb507d05d6 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp @@ -14,6 +14,11 @@ using namespace nx::core; using namespace nx::core::OrientationUtilities; +namespace +{ +constexpr usize k_ChunkSize = 4096; +} // namespace + // ----------------------------------------------------------------------------- ComputeAvgCAxes::ComputeAvgCAxes(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeAvgCAxesInputValues* inputValues) : m_DataStructure(dataStructure) @@ -35,14 +40,18 @@ const std::atomic_bool& ComputeAvgCAxes::getCancel() // ----------------------------------------------------------------------------- Result<> ComputeAvgCAxes::operator()() { + // Cache ensemble-level crystal structures locally to avoid per-element OOC overhead + const auto& crystalStructuresStoreRef = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); + const usize numCrystalStructures = crystalStructuresStoreRef.getSize(); + auto crystalStructuresCache = std::make_unique(numCrystalStructures); + crystalStructuresStoreRef.copyIntoBuffer(0, nonstd::span(crystalStructuresCache.get(), numCrystalStructures)); // Figure out if all phases are either Hexagonal-Low 6/m or Hexagonal-High 6/mmm Laue Phases - const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); bool allPhasesHexagonal = true; bool noPhasesHexagonal = true; - for(usize i = 1; i < crystalStructures.size(); ++i) + for(usize i = 1; i < numCrystalStructures; ++i) { - const auto crystalStructureType = crystalStructures[i]; + const auto crystalStructureType = crystalStructuresCache[i]; const bool isHex = crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_High || crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_Low; allPhasesHexagonal = allPhasesHexagonal && isHex; noPhasesHexagonal = noPhasesHexagonal && !isHex; @@ -62,87 +71,107 @@ Result<> ComputeAvgCAxes::operator()() result.warnings().push_back({-76403, "Non Hexagonal phases were found. All calculations for non Hexagonal phases will be skipped and a NaN value inserted."}); } - const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); - const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); - const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); - auto& avgCAxes = m_DataStructure.getDataRefAs(m_InputValues->AvgCAxesArrayPath); + const auto& featureIdsStoreRef = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + const auto& quatsStoreRef = m_DataStructure.getDataAs(m_InputValues->QuatsArrayPath)->getDataStoreRef(); + const auto& cellPhasesStoreRef = m_DataStructure.getDataAs(m_InputValues->CellPhasesArrayPath)->getDataStoreRef(); + auto& avgCAxesStoreRef = m_DataStructure.getDataAs(m_InputValues->AvgCAxesArrayPath)->getDataStoreRef(); + + const usize totalPoints = featureIdsStoreRef.getNumberOfTuples(); + const usize totalFeatures = avgCAxesStoreRef.getNumberOfTuples(); - const usize totalPoints = featureIds.getNumberOfTuples(); - const usize totalFeatures = avgCAxes.getNumberOfTuples(); + // Cache feature-level avgCAxes locally (random access by featureId in the hot loop) + const usize avgCAxesElements = totalFeatures * 3; + auto avgCAxesCache = std::make_unique(avgCAxesElements); + avgCAxesStoreRef.copyIntoBuffer(0, nonstd::span(avgCAxesCache.get(), avgCAxesElements)); const Eigen::Vector3d cAxis{0.0f, 0.0f, 1.0f}; Eigen::Vector3d c1{0.0f, 0.0f, 0.0f}; - std::vector counter(totalFeatures, 0); + auto counter = std::make_unique(totalFeatures); - // Loop over each cell - for(usize i = 0; i < totalPoints; i++) + // Allocate chunk buffers for cell-level arrays (sequential access) + auto featureIdsChunk = std::make_unique(k_ChunkSize); + auto cellPhasesChunk = std::make_unique(k_ChunkSize); + auto quatsChunk = std::make_unique(k_ChunkSize * 4); + + // Loop over each cell in chunks to minimize OOC overhead + usize tupleIdx = 0; + while(tupleIdx < totalPoints) { if(m_ShouldCancel) { - return {}; + return result; } - int32 currentFeatureId = featureIds[i]; - // If the featureId for a given cell is valid ( > 0) then analyze that value - if(currentFeatureId > 0) - { - const int32 currentCellPhase = cellPhases[i]; // Get the current cell phase - const auto crystalStructureType = crystalStructures[currentCellPhase]; // Get the CrystalStructure, i.e., Laue class of the cell - const usize cAxesIndex = 3 * currentFeatureId; - - // Ensure the Laue class is correct, otherwise mark the values with a NaN and continue - if(crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_High && crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_Low) - { - avgCAxes[cAxesIndex] = NAN; - avgCAxes[cAxesIndex + 1] = NAN; - avgCAxes[cAxesIndex + 2] = NAN; - continue; - } - - counter[currentFeatureId]++; // Increment the count - const usize quatIndex = i * 4; + const usize chunkTuples = std::min(k_ChunkSize, totalPoints - tupleIdx); - // Create the 3x3 Orientation Matrix from the Quaternion. This represents a passive rotation matrix - ebsdlib::OrientationMatrixDType oMatrix = ebsdlib::QuaternionDType(quats[quatIndex], quats[quatIndex + 1], quats[quatIndex + 2], quats[quatIndex + 3]).toOrientationMatrix(); + // Bulk-read cell-level data for this chunk + featureIdsStoreRef.copyIntoBuffer(tupleIdx, nonstd::span(featureIdsChunk.get(), chunkTuples)); + cellPhasesStoreRef.copyIntoBuffer(tupleIdx, nonstd::span(cellPhasesChunk.get(), chunkTuples)); + quatsStoreRef.copyIntoBuffer(tupleIdx * 4, nonstd::span(quatsChunk.get(), chunkTuples * 4)); - // Convert the passive rotation matrix to an active rotation matrix by taking the transpose - // Multiply the active transformation matrix by the C-Axis (as Miller Index). This actively rotates - // the crystallographic C-Axis (which is along the <0,0,1> direction) into the physical sample - // reference frame - c1 = oMatrix.transpose() * cAxis; - - // normalize so that the magnitude is 1 - c1.normalize(); - - // Compute the running average c-axis and normalize the result - Eigen::Vector3d curCAxis{0.0f, 0.0f, 0.0f}; - curCAxis[0] = avgCAxes[cAxesIndex] / static_cast(counter[currentFeatureId]); - curCAxis[1] = avgCAxes[cAxesIndex + 1] / static_cast(counter[currentFeatureId]); - curCAxis[2] = avgCAxes[cAxesIndex + 2] / static_cast(counter[currentFeatureId]); - curCAxis.normalize(); - - // Ensure that angle between the current point's sample reference frame C-Axis - // and the running average sample C-Axis is positive - float64 w = ImageRotationUtilities::CosBetweenVectors(c1, curCAxis); - if(w < 0.0) + for(usize t = 0; t < chunkTuples; t++) + { + const int32 currentFeatureId = featureIdsChunk[t]; + // If the featureId for a given cell is valid ( > 0) then analyze that value + if(currentFeatureId > 0) { - c1 *= -1.0f; + const int32 currentCellPhase = cellPhasesChunk[t]; // Get the current cell phase + const auto crystalStructureType = crystalStructuresCache[currentCellPhase]; // Get the CrystalStructure, i.e., Laue class of the cell + const usize cAxesIndex = 3 * static_cast(currentFeatureId); + + // Ensure the Laue class is correct, otherwise mark the values with a NaN and continue + if(crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_High && crystalStructureType != ebsdlib::CrystalStructure::Hexagonal_Low) + { + avgCAxesCache[cAxesIndex] = NAN; + avgCAxesCache[cAxesIndex + 1] = NAN; + avgCAxesCache[cAxesIndex + 2] = NAN; + continue; + } + + counter[currentFeatureId]++; // Increment the count + const usize quatOffset = t * 4; + + // Create the 3x3 Orientation Matrix from the Quaternion. This represents a passive rotation matrix + ebsdlib::OrientationMatrixDType oMatrix = + ebsdlib::QuaternionDType(quatsChunk[quatOffset], quatsChunk[quatOffset + 1], quatsChunk[quatOffset + 2], quatsChunk[quatOffset + 3]).toOrientationMatrix(); + + // Convert the passive rotation matrix to an active rotation matrix by taking the transpose + // Multiply the active transformation matrix by the C-Axis (as Miller Index). This actively rotates + // the crystallographic C-Axis (which is along the <0,0,1> direction) into the physical sample + // reference frame + c1 = oMatrix.transpose() * cAxis; + + // normalize so that the magnitude is 1 + c1.normalize(); + + // Compute the running average c-axis and normalize the result + Eigen::Vector3d curCAxis{0.0f, 0.0f, 0.0f}; + curCAxis[0] = avgCAxesCache[cAxesIndex] / static_cast(counter[currentFeatureId]); + curCAxis[1] = avgCAxesCache[cAxesIndex + 1] / static_cast(counter[currentFeatureId]); + curCAxis[2] = avgCAxesCache[cAxesIndex + 2] / static_cast(counter[currentFeatureId]); + curCAxis.normalize(); + + // Ensure that angle between the current point's sample reference frame C-Axis + // and the running average sample C-Axis is positive + float64 w = ImageRotationUtilities::CosBetweenVectors(c1, curCAxis); + if(w < 0.0) + { + c1 *= -1.0f; + } + + // Continue summing up the rotations + avgCAxesCache[cAxesIndex] += static_cast(c1[0]); + avgCAxesCache[cAxesIndex + 1] += static_cast(c1[1]); + avgCAxesCache[cAxesIndex + 2] += static_cast(c1[2]); } - - // Continue summing up the rotations - float value = avgCAxes[cAxesIndex] + c1[0]; - avgCAxes[cAxesIndex] = value; - - value = avgCAxes[cAxesIndex + 1] + c1[1]; - avgCAxes[cAxesIndex + 1] = value; - - value = avgCAxes[cAxesIndex + 2] + c1[2]; - avgCAxes[cAxesIndex + 2] = value; } + + tupleIdx += chunkTuples; } - for(size_t i = 1; i < totalFeatures; i++) + // Normalize the accumulated c-axis values + for(usize i = 1; i < totalFeatures; i++) { if(m_ShouldCancel) { @@ -150,7 +179,7 @@ Result<> ComputeAvgCAxes::operator()() } const usize tupleIndex = i * 3; - float32 avgCAxesValue = avgCAxes[tupleIndex]; + float32 avgCAxesValue = avgCAxesCache[tupleIndex]; if(std::isnan(avgCAxesValue)) { continue; @@ -159,25 +188,20 @@ Result<> ComputeAvgCAxes::operator()() // masked out? Maybe? if(counter[i] == 0) { - avgCAxes[tupleIndex] = 0; - avgCAxes[tupleIndex + 1] = 0; - avgCAxes[tupleIndex + 2] = 1; + avgCAxesCache[tupleIndex] = 0; + avgCAxesCache[tupleIndex + 1] = 0; + avgCAxesCache[tupleIndex + 2] = 1; } else { - // Compute the final average c-axis value - float value = avgCAxes[3 * i]; - value /= static_cast(counter[i]); - avgCAxes[3 * i] = value; - - value = avgCAxes[3 * i + 1]; - value /= static_cast(counter[i]); - avgCAxes[3 * i + 1] = value; - - value = avgCAxes[3 * i + 2]; - value /= static_cast(counter[i]); - avgCAxes[3 * i + 2] = value; + avgCAxesCache[tupleIndex] /= static_cast(counter[i]); + avgCAxesCache[tupleIndex + 1] /= static_cast(counter[i]); + avgCAxesCache[tupleIndex + 2] /= static_cast(counter[i]); } } + + // Write cached avgCAxes data back to the store + avgCAxesStoreRef.copyFromBuffer(0, nonstd::span(avgCAxesCache.get(), avgCAxesElements)); + return result; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp index 40b9b53e0c..bdd1998c7d 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp @@ -8,7 +8,9 @@ #include #include -#include +#include + +#include using namespace nx::core; @@ -194,6 +196,7 @@ void UpdateEulerArray(AbstractDataStore& eulerArray, const ebsdlib::Euler& eulerArray.setValue(tupleIndex * 3 + 2, euler[2]); } +constexpr usize k_ChunkTuples = 65536; } // namespace // ----------------------------------------------------------------------------- @@ -368,90 +371,122 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() { std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - Int32Array& featureIds = m_DataStructure.getDataRefAs(m_InputValues->cellFeatureIdsArrayPath); - Int32Array& phases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); - Float32Array& quats = m_DataStructure.getDataRefAs(m_InputValues->cellQuatsArrayPath); + auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->cellFeatureIdsArrayPath); + auto& phases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); + auto& quats = m_DataStructure.getDataRefAs(m_InputValues->cellQuatsArrayPath); + + auto& crystalStructuresArray = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); - UInt32Array& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); + auto& avgQuatsStore = m_DataStructure.getDataRefAs(m_InputValues->avgQuatsArrayPath).getDataStoreRef(); + auto& avgEulerStore = m_DataStructure.getDataRefAs(m_InputValues->avgEulerAnglesArrayPath).getDataStoreRef(); - auto& avgQuats = m_DataStructure.getDataRefAs(m_InputValues->avgQuatsArrayPath).getDataStoreRef(); - auto& avgEuler = m_DataStructure.getDataRefAs(m_InputValues->avgEulerAnglesArrayPath).getDataStoreRef(); + const usize totalPoints = featureIds.getNumberOfTuples(); - const size_t totalPoints = featureIds.getNumberOfTuples(); + const usize totalFeatures = avgQuatsStore.getNumberOfTuples(); + std::vector counts(totalFeatures, 0.0f); - size_t totalFeatures = avgQuats.getNumberOfTuples(); - std::vector counts(totalFeatures, 0.0f); + // Cache crystal structures locally (ensemble-level, tiny) + const usize numPhases = crystalStructuresArray.getNumberOfTuples(); + std::vector crystalStructures(numPhases); + crystalStructuresArray.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructures.data(), numPhases)); - // initialize the output arrays - avgQuats.fill(0.0F); - // Initialize all Euler Angles to Zero - avgEuler.fill(0.0F); + // Local cache for avgQuats (feature-level, manageable) + std::vector localAvgQuats(totalFeatures * 4, 0.0f); // Get the Identity Quaternion static const ebsdlib::QuatF identityQuat(0.0f, 0.0f, 0.0f, 1.0f); - for(size_t i = 0; i < totalPoints; i++) + // Chunked accumulation of cell-level data + const auto& featureIdsStore = featureIds.getDataStoreRef(); + const auto& phasesStore = phases.getDataStoreRef(); + const auto& quatsStore = quats.getDataStoreRef(); + + auto featureIdBuf = std::make_unique(k_ChunkTuples); + auto phasesBuf = std::make_unique(k_ChunkTuples); + auto quatsBuf = std::make_unique(k_ChunkTuples * 4); + + for(usize offset = 0; offset < totalPoints; offset += k_ChunkTuples) { if(m_ShouldCancel) { return {}; } - const int32_t currentFeatureId = featureIds[i]; - const int32_t currentPhase = phases[i]; - // As long as we have a valid `currentPhase` value which is used as an index - // into the CrystalStructures array. We ALWAYS ignore the first value in the - // CrystalStructures array. So therefore the `currentPhase` MUST be > 0. - // We can use `currentFeatureId = 0` because if someone is just wanting to compute - // the average of a bunch of orientations they may have labeled the "FeatureIds = 0" - // for all the values. The most important value is the `currentPhase` which for - // the grand majority of historical data should be > 0. - // Now in theory someone could absolutely manually import data into an "Ensemble" - // Array and NOT have the zero index as `unknown` in which case this check will - // fail them and they will not compute anything most likely. The documentation - // for the filter should be updated to cover these use-cases. - if(currentPhase > 0) - { - const uint32 xtal = crystalStructures[currentPhase]; - counts[currentFeatureId] += 1.0f; - ebsdlib::QuatF voxQuat(quats[i * 4], quats[i * 4 + 1], quats[i * 4 + 2], quats[i * 4 + 3]); - ebsdlib::QuatF curAvgQuat(avgQuats[currentFeatureId * 4], avgQuats[currentFeatureId * 4 + 1], avgQuats[currentFeatureId * 4 + 2], avgQuats[currentFeatureId * 4 + 3]); - ebsdlib::QuatF finalAvgQuat(avgQuats[currentFeatureId * 4], avgQuats[currentFeatureId * 4 + 1], avgQuats[currentFeatureId * 4 + 2], avgQuats[currentFeatureId * 4 + 3]); - curAvgQuat = curAvgQuat.scalarDivide(counts[currentFeatureId]); + const usize count = std::min(k_ChunkTuples, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), count)); + phasesStore.copyIntoBuffer(offset, nonstd::span(phasesBuf.get(), count)); + quatsStore.copyIntoBuffer(offset * 4, nonstd::span(quatsBuf.get(), count * 4)); - if(counts[currentFeatureId] == 1.0f) + for(usize i = 0; i < count; i++) + { + const int32 currentFeatureId = featureIdBuf[i]; + const int32 currentPhase = phasesBuf[i]; + if(currentFeatureId > 0 && currentPhase > 0) { - curAvgQuat = ebsdlib::QuatF::identity(); - } - voxQuat = orientationOps[xtal]->getNearestQuat(curAvgQuat, voxQuat); - curAvgQuat = finalAvgQuat + voxQuat; + const uint32 xtal = crystalStructures[currentPhase]; + counts[currentFeatureId] += 1.0f; + + const usize qi = i * 4; + ebsdlib::QuatF voxQuat(quatsBuf[qi], quatsBuf[qi + 1], quatsBuf[qi + 2], quatsBuf[qi + 3]); + + const usize fi = static_cast(currentFeatureId) * 4; + ebsdlib::QuatF curAvgQuat(localAvgQuats[fi], localAvgQuats[fi + 1], localAvgQuats[fi + 2], localAvgQuats[fi + 3]); + ebsdlib::QuatF finalAvgQuat = curAvgQuat; + + curAvgQuat = curAvgQuat.scalarDivide(counts[currentFeatureId]); + + if(counts[currentFeatureId] == 1.0f) + { + curAvgQuat = ebsdlib::QuatF::identity(); + } + voxQuat = orientationOps[xtal]->getNearestQuat(curAvgQuat, voxQuat); + curAvgQuat = finalAvgQuat + voxQuat; - UpdateQuaternionArray(avgQuats, curAvgQuat, currentFeatureId); + localAvgQuats[fi] = curAvgQuat.x(); + localAvgQuats[fi + 1] = curAvgQuat.y(); + localAvgQuats[fi + 2] = curAvgQuat.z(); + localAvgQuats[fi + 3] = curAvgQuat.w(); + } } } - for(size_t featureId = 0; featureId < totalFeatures; featureId++) + // Second pass: normalize and convert to Euler angles (feature-level only) + std::vector localAvgEuler(totalFeatures * 3, 0.0f); + + for(usize featureId = 1; featureId < totalFeatures; featureId++) { if(m_ShouldCancel) { return {}; } + const usize fi = featureId * 4; if(counts[featureId] == 0.0f) { - UpdateQuaternionArray(avgQuats, identityQuat, featureId); - continue; + localAvgQuats[fi] = identityQuat.x(); + localAvgQuats[fi + 1] = identityQuat.y(); + localAvgQuats[fi + 2] = identityQuat.z(); + localAvgQuats[fi + 3] = identityQuat.w(); } - ebsdlib::QuatF curAvgQuat(avgQuats[featureId * 4], avgQuats[featureId * 4 + 1], avgQuats[featureId * 4 + 2], avgQuats[featureId * 4 + 3]); + ebsdlib::QuatF curAvgQuat(localAvgQuats[fi], localAvgQuats[fi + 1], localAvgQuats[fi + 2], localAvgQuats[fi + 3]); curAvgQuat = curAvgQuat.scalarDivide(counts[featureId]); - curAvgQuat = curAvgQuat.normalize().getPositiveOrientation(); // Be sure the Quaterion is in the Northern Hemisphere - UpdateQuaternionArray(avgQuats, curAvgQuat, featureId); + curAvgQuat = curAvgQuat.normalize().getPositiveOrientation(); + localAvgQuats[fi] = curAvgQuat.x(); + localAvgQuats[fi + 1] = curAvgQuat.y(); + localAvgQuats[fi + 2] = curAvgQuat.z(); + localAvgQuats[fi + 3] = curAvgQuat.w(); - // Update the value for the average Euler. ebsdlib::EulerFType eu = ebsdlib::QuaternionFType(curAvgQuat).toEuler(); - UpdateEulerArray(avgEuler, eu, featureId); + const usize ei = featureId * 3; + localAvgEuler[ei] = eu[0]; + localAvgEuler[ei + 1] = eu[1]; + localAvgEuler[ei + 2] = eu[2]; } + // Write feature-level results back to DataStore + avgQuatsStore.copyFromBuffer(0, nonstd::span(localAvgQuats.data(), localAvgQuats.size())); + avgEulerStore.copyFromBuffer(0, nonstd::span(localAvgEuler.data(), localAvgEuler.size())); + return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp index 62cbfdb3af..083ecbdac7 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp @@ -56,51 +56,69 @@ Result<> ComputeCAxisLocations::operator()() "skipped and a NaN value inserted."}); } - std::vector m_OrientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - const auto& quaternions = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); auto& cAxisLocation = m_DataStructure.getDataRefAs(m_InputValues->CAxisLocationsArrayName); const usize totalPoints = quaternions.getNumberOfTuples(); + // Cache ensemble-level crystal structures into local vector + const usize numPhases = crystalStructures.getNumberOfTuples(); + std::vector crystalStructuresBuf(numPhases); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresBuf.data(), numPhases)); + + // Process cells in chunks using bulk I/O + constexpr usize k_ChunkSize = 65536; const Eigen::Vector3f cAxis{0.0f, 0.0f, 1.0f}; Eigen::Vector3f c1{0.0f, 0.0f, 0.0f}; - usize index = 0; - for(size_t i = 0; i < totalPoints; i++) + auto& quatStore = quaternions.getDataStoreRef(); + auto& phaseStore = cellPhases.getDataStoreRef(); + auto& outputStore = cAxisLocation.getDataStoreRef(); + + for(usize chunkStart = 0; chunkStart < totalPoints; chunkStart += k_ChunkSize) { if(m_ShouldCancel) { return {}; } - index = 3 * i; - const auto crystalStructureType = crystalStructures[cellPhases[i]]; - if(crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_High || crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_Low) + const usize chunkCount = std::min(k_ChunkSize, totalPoints - chunkStart); + + // Bulk-read quaternions (4 components), phases (1 component) + std::vector quatBuf(chunkCount * 4); + std::vector phaseBuf(chunkCount); + std::vector outputBuf(chunkCount * 3); + + quatStore.copyIntoBuffer(chunkStart * 4, nonstd::span(quatBuf.data(), chunkCount * 4)); + phaseStore.copyIntoBuffer(chunkStart, nonstd::span(phaseBuf.data(), chunkCount)); + + for(usize i = 0; i < chunkCount; i++) { - const usize quatIndex = i * 4; - ebsdlib::OrientationMatrixFType oMatrix = - ebsdlib::QuaternionFType(quaternions[quatIndex], quaternions[quatIndex + 1], quaternions[quatIndex + 2], quaternions[quatIndex + 3]).toOrientationMatrix(); - // transpose the g matrices so when c-axis is multiplied by it - // it will give the sample direction that the c-axis is along - c1 = oMatrix.transpose() * cAxis; - // normalize so that the magnitude is 1 - c1.normalize(); - if(c1[2] < 0) + const auto crystalStructureType = crystalStructuresBuf[phaseBuf[i]]; + if(crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_High || crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_Low) { - c1 *= -1.0f; + const usize qi = i * 4; + ebsdlib::OrientationMatrixFType oMatrix = ebsdlib::QuaternionFType(quatBuf[qi], quatBuf[qi + 1], quatBuf[qi + 2], quatBuf[qi + 3]).toOrientationMatrix(); + c1 = oMatrix.transpose() * cAxis; + c1.normalize(); + if(c1[2] < 0) + { + c1 *= -1.0f; + } + outputBuf[i * 3] = c1[0]; + outputBuf[i * 3 + 1] = c1[1]; + outputBuf[i * 3 + 2] = c1[2]; + } + else + { + outputBuf[i * 3] = NAN; + outputBuf[i * 3 + 1] = NAN; + outputBuf[i * 3 + 2] = NAN; } - cAxisLocation[index] = c1[0]; - cAxisLocation[index + 1] = c1[1]; - cAxisLocation[index + 2] = c1[2]; - } - else - { - cAxisLocation[index] = NAN; - cAxisLocation[index + 1] = NAN; - cAxisLocation[index + 2] = NAN; } + + outputStore.copyFromBuffer(chunkStart * 3, nonstd::span(outputBuf.data(), chunkCount * 3)); } return result; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp index 6fb705cd8a..03f9d869af 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp @@ -29,12 +29,19 @@ ComputeFeatureNeighborCAxisMisalignments::~ComputeFeatureNeighborCAxisMisalignme // ----------------------------------------------------------------------------- Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() { + // ------------------------------------------------------------------------- + // Cache ensemble-level crystalStructures locally (tiny array) + // ------------------------------------------------------------------------- + const auto& crystalStructuresStore = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); + const usize numPhases = crystalStructuresStore.getNumberOfTuples(); + std::vector crystalStructures(numPhases); + crystalStructuresStore.copyIntoBuffer(0, nonstd::span(crystalStructures.data(), numPhases)); + // Validate any Crystal Structure issues early in the process. // If none of the phases are hexagonal, then report and return - const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); bool allPhasesHexagonal = true; bool noPhasesHexagonal = true; - for(usize i = 1; i < crystalStructures.size(); ++i) + for(usize i = 1; i < numPhases; ++i) { const auto crystalStructureType = crystalStructures[i]; const bool isHex = crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_High || crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_Low; @@ -55,22 +62,35 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() result.warnings().push_back({-1563, "Non Hexagonal phases were found. All calculations for non Hexagonal phases will be skipped and a NaN value inserted."}); } - // Get references to all the input data + // ------------------------------------------------------------------------- + // Cache feature-level arrays locally (O(features) — thousands, not millions) + // ------------------------------------------------------------------------- + const auto& featurePhasesStore = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); + const usize totalFeatures = featurePhasesStore.getNumberOfTuples(); + std::vector featurePhases(totalFeatures); + featurePhasesStore.copyIntoBuffer(0, nonstd::span(featurePhases.data(), totalFeatures)); + + const auto& avgQuatsStore = m_DataStructure.getDataAs(m_InputValues->AvgQuatsArrayPath)->getDataStoreRef(); + const usize numQuatComps = avgQuatsStore.getNumberOfComponents(); + const usize quatSize = totalFeatures * numQuatComps; + std::vector featureAvgQuat(quatSize); + avgQuatsStore.copyIntoBuffer(0, nonstd::span(featureAvgQuat.data(), quatSize)); + + // Get references to all the input/output data auto& neighborList = m_DataStructure.getDataRefAs>(m_InputValues->NeighborListArrayPath); - const auto& featurePhases = m_DataStructure.getDataRefAs(m_InputValues->FeaturePhasesArrayPath); - const auto& featureAvgQuat = m_DataStructure.getDataRefAs(m_InputValues->AvgQuatsArrayPath); - - // Get references to all the Output data auto& cAxisMisalignmentList = m_DataStructure.getDataRefAs>(m_InputValues->CAxisMisalignmentListArrayName); + + // ------------------------------------------------------------------------- + // Output buffer for avgCAxisMisalignment — accumulate locally, bulk-write at end + // ------------------------------------------------------------------------- Float32Array* avgCAxisMisalignmentPtr = nullptr; + std::vector avgCAxisBuf; if(m_InputValues->FindAvgMisals) { avgCAxisMisalignmentPtr = m_DataStructure.getDataAs(m_InputValues->AvgCAxisMisalignmentsArrayName); + avgCAxisBuf.resize(totalFeatures, 0.0f); } - const usize totalFeatures = featurePhases.getNumberOfTuples(); - const usize numQuatComps = featureAvgQuat.getNumberOfComponents(); - std::vector> misalignmentLists(totalFeatures); const Eigen::Vector3d cAxis{0.0, 0.0, 1.0}; @@ -85,7 +105,6 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() { return {}; } - // Get the crystal structure of phase 1 xtalPhase1 = crystalStructures[featurePhases[featureIdx]]; @@ -139,8 +158,7 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() // If we are finding the average misorientation, then start accumulating those values if(m_InputValues->FindAvgMisals) { - float32 value = avgCAxisMisalignmentPtr->getValue(featureIdx) + currentMisalignmentList[j]; - avgCAxisMisalignmentPtr->setValue(featureIdx, value); + avgCAxisBuf[featureIdx] += currentMisalignmentList[j]; } } else // The current feature and it's neighbor do not match in crystal structures so place a NaN value @@ -159,12 +177,11 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() { if(hexNeighborListSize > 0) { - double value = avgCAxisMisalignmentPtr->getValue(featureIdx) / static_cast(hexNeighborListSize); - avgCAxisMisalignmentPtr->setValue(featureIdx, value); + avgCAxisBuf[featureIdx] = static_cast(static_cast(avgCAxisBuf[featureIdx]) / static_cast(hexNeighborListSize)); } else { - avgCAxisMisalignmentPtr->setValue(featureIdx, std::nan("")); + avgCAxisBuf[featureIdx] = std::nanf(""); } hexNeighborListSize = 0; } @@ -172,5 +189,13 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() cAxisMisalignmentList.setList(featureIdx, {currentMisalignmentList.begin(), currentMisalignmentList.end()}); } + // ------------------------------------------------------------------------- + // Bulk-write the avgCAxisMisalignment output buffer back to the DataStore + // ------------------------------------------------------------------------- + if(m_InputValues->FindAvgMisals) + { + avgCAxisMisalignmentPtr->getDataStoreRef().copyFromBuffer(0, nonstd::span(avgCAxisBuf.data(), totalFeatures)); + } + return result; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp index 9ab8032387..518aff28a1 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp @@ -12,7 +12,10 @@ #include #include +#include + #include +#include using namespace nx::core; using namespace nx::core::OrientationUtilities; @@ -35,15 +38,19 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() { /* ************************************************************************** - * This section performs a sanity check to ensure that at least 1 phase is - * hexagonal. + * Cache ensemble-level crystalStructures locally (tiny array, avoids + * per-element OOC overhead during the main cell loop) */ const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + const usize numCrystalStructures = crystalStructures.getNumberOfTuples(); + std::vector crystalStructuresLocal(numCrystalStructures); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresLocal.data(), numCrystalStructures)); + bool allPhasesHexagonal = true; bool noPhasesHexagonal = true; - for(usize i = 1; i < crystalStructures.size(); ++i) + for(usize i = 1; i < numCrystalStructures; ++i) { - const auto crystalStructureType = crystalStructures[i]; + const auto crystalStructureType = crystalStructuresLocal[i]; const bool isHex = crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_High || crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_Low; allPhasesHexagonal = allPhasesHexagonal && isHex; noPhasesHexagonal = noPhasesHexagonal && !isHex; @@ -64,27 +71,30 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() } /* ************************************************************************** - * Get References to the Input and output Data Arrays + * Get DataStore references for bulk I/O */ - // Input Cell Data - const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); - const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); - const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); - // Input Feature Data + // Input Cell Data (DataStore refs for copyIntoBuffer) + const auto& featureIdsStore = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath).getDataStoreRef(); + const auto& quatsStore = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath).getDataStoreRef(); + const auto& cellPhasesStore = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath).getDataStoreRef(); + + // Input Feature Data — cache avgCAxes locally (feature-level, small) const auto& avgCAxes = m_DataStructure.getDataRefAs(m_InputValues->AvgCAxesArrayPath); + const usize totalFeatures = avgCAxes.getNumberOfTuples(); + const usize avgCAxesSize = totalFeatures * 3; + std::vector avgCAxesLocal(avgCAxesSize); + avgCAxes.getDataStoreRef().copyIntoBuffer(0, nonstd::span(avgCAxesLocal.data(), avgCAxesSize)); + + // Output Cell Data (DataStore ref for copyFromBuffer) + auto& cellRefCAxisMisStore = m_DataStructure.getDataRefAs(m_InputValues->FeatureReferenceCAxisMisorientationsArrayPath).getDataStoreRef(); - // Output Cell Data - auto& cellRefCAxisMis = m_DataStructure.getDataRefAs(m_InputValues->FeatureReferenceCAxisMisorientationsArrayPath); // Output Feature Data auto& featAvgCAxisMis = m_DataStructure.getDataRefAs(m_InputValues->FeatureAvgCAxisMisorientationsArrayPath); featAvgCAxisMis.fill(0.0f); auto& featStdevCAxisMis = m_DataStructure.getDataRefAs(m_InputValues->FeatureStdevCAxisMisorientationsArrayPath); featStdevCAxisMis.fill(0.0f); - const usize totalPoints = featureIds.getNumberOfTuples(); - const usize totalFeatures = avgCAxes.getNumberOfTuples(); - - const usize numQuatComps = quats.getNumberOfComponents(); + const usize numQuatComps = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath).getNumberOfComponents(); std::vector counts(totalFeatures, 0ULL); std::vector avgMisorientations(totalFeatures, 0.0f); @@ -94,11 +104,21 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() const auto xPoints = static_cast(uDims[0]); const auto yPoints = static_cast(uDims[1]); const auto zPoints = static_cast(uDims[2]); + const usize sliceSize = static_cast(xPoints * yPoints); + const usize quatSliceSize = sliceSize * numQuatComps; const Eigen::Vector3d cAxis{0.0, 0.0, 1.0}; + // Z-slice buffers for cell-level arrays (avoids per-element OOC access) + std::vector featureIdSlice(sliceSize); + std::vector cellPhaseSlice(sliceSize); + std::vector quatSlice(quatSliceSize); + std::vector outputSlice(sliceSize, 0.0f); + /* ************************************************************************** - * Loop over all cells in the ImageGeometry + * Loop over all cells in the ImageGeometry, one Z-slice at a time. + * Each slice is bulk-read from the DataStore, processed, and the output + * is bulk-written back. */ for(int64 plane = 0; plane < zPoints; plane++) { @@ -106,17 +126,23 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() { return {}; } + const usize sliceOffset = static_cast(plane) * sliceSize; + + // Bulk-read this Z-slice of input cell data + featureIdsStore.copyIntoBuffer(sliceOffset, nonstd::span(featureIdSlice.data(), sliceSize)); + cellPhasesStore.copyIntoBuffer(sliceOffset, nonstd::span(cellPhaseSlice.data(), sliceSize)); + quatsStore.copyIntoBuffer(sliceOffset * numQuatComps, nonstd::span(quatSlice.data(), quatSliceSize)); for(int64 row = 0; row < yPoints; row++) { for(int64 col = 0; col < xPoints; col++) { - int64 cellIdx = (plane * xPoints * yPoints) + (row * xPoints) + col; - const usize quatTupleIndex = cellIdx * numQuatComps; - const uint32 crystalStructureType = crystalStructures[cellPhases[cellIdx]]; + const usize localIdx = static_cast(row * xPoints + col); + const usize quatLocalIdx = localIdx * numQuatComps; + const int32 cellFeatureId = featureIdSlice[localIdx]; + const int32 cellPhase = cellPhaseSlice[localIdx]; + const uint32 crystalStructureType = crystalStructuresLocal[cellPhase]; const bool isHex = crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_High || crystalStructureType == ebsdlib::CrystalStructure::Hexagonal_Low; - int32_t cellFeatureId = featureIds[cellIdx]; - int32_t cellPhase = cellPhases[cellIdx]; // Make sure the cell is Hexagonal Laue class, the featureId and phases are valid // INVALID featureIds have a value of ZERO @@ -125,7 +151,7 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() { // Create the OrientationMatrix from the Quaternion ebsdlib::OrientationMatrixDType oMatrix = - ebsdlib::QuaternionDType(quats[quatTupleIndex], quats[quatTupleIndex + 1], quats[quatTupleIndex + 2], quats[quatTupleIndex + 3]).toOrientationMatrix(); + ebsdlib::QuaternionDType(quatSlice[quatLocalIdx], quatSlice[quatLocalIdx + 1], quatSlice[quatLocalIdx + 2], quatSlice[quatLocalIdx + 3]).toOrientationMatrix(); // Transpose the OM and multiply by cAxis to rotate cAxis Eigen::Vector3d c1 = oMatrix.transpose() * cAxis; @@ -133,7 +159,8 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() c1.normalize(); // normalize the features average C-Axis - Eigen::Vector3d avgCAxisMis = {avgCAxes[3 * cellFeatureId], avgCAxes[3 * cellFeatureId + 1], avgCAxes[3 * cellFeatureId + 2]}; + const usize avgCAxesIdx = static_cast(cellFeatureId) * 3; + Eigen::Vector3d avgCAxisMis = {avgCAxesLocal[avgCAxesIdx], avgCAxesLocal[avgCAxesIdx + 1], avgCAxesLocal[avgCAxesIdx + 2]}; avgCAxisMis.normalize(); // Calculate the angle between the current C-Axis and the Feature's Average C-Axis @@ -146,16 +173,19 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() w = 180.0 - w; } - cellRefCAxisMis.setValue(cellIdx, static_cast(w)); + outputSlice[localIdx] = static_cast(w); counts[cellFeatureId]++; avgMisorientations[cellFeatureId] += static_cast(w); } else { - cellRefCAxisMis.setValue(cellIdx, 0.0f); + outputSlice[localIdx] = 0.0f; } } } + + // Bulk-write this Z-slice of output cell data + cellRefCAxisMisStore.copyFromBuffer(sliceOffset, nonstd::span(outputSlice.data(), sliceSize)); } // Loop over all the features from the feature attribute matrix and compute the @@ -177,18 +207,24 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() } // These 2 loops compute the population standard deviation of those misorientations for - // each feature. + // each feature. Re-read cell data one Z-slice at a time. std::vector stdevs(totalFeatures, 0.0); - for(usize cellIdx = 0; cellIdx < totalPoints; cellIdx++) + for(int64 plane = 0; plane < zPoints; plane++) { if(m_ShouldCancel) { return {}; } + const usize sliceOffset = static_cast(plane) * sliceSize; + featureIdsStore.copyIntoBuffer(sliceOffset, nonstd::span(featureIdSlice.data(), sliceSize)); + cellRefCAxisMisStore.copyIntoBuffer(sliceOffset, nonstd::span(outputSlice.data(), sliceSize)); - const int32 featureId = featureIds[cellIdx]; - double diff = cellRefCAxisMis.getValue(cellIdx) - featAvgCAxisMis.getValue(featureId); - stdevs[featureId] += (diff * diff); + for(usize localIdx = 0; localIdx < sliceSize; localIdx++) + { + const int32 featureId = featureIdSlice[localIdx]; + double diff = outputSlice[localIdx] - featAvgCAxisMis.getValue(featureId); + stdevs[featureId] += (diff * diff); + } } // Finish computing the standard deviation in this loop diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp index c40533ba98..3e21161b28 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp @@ -9,8 +9,17 @@ #include +#include + +#include + using namespace nx::core; +namespace +{ +constexpr usize k_ChunkTuples = 65536; +} // namespace + // ----------------------------------------------------------------------------- ComputeFeatureReferenceMisorientations::ComputeFeatureReferenceMisorientations(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureReferenceMisorientationsInputValues* inputValues) @@ -41,15 +50,10 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); - // Get the average quats data array. It will be null unless m_InputValues->ReferenceOrientation = 0 const auto* avgQuatsPtr = m_DataStructure.getDataAs(m_InputValues->AvgQuatsArrayPath); - - // Get the Feature AttributeMatrix. It will be null unless m_InputValues->ReferenceOrientation = 1 const auto* featureAttrMatPtr = m_DataStructure.getDataAs(m_InputValues->FeatureAttributeMatrixPath); - const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); - // Output Arrays auto& featureReferenceMisorientations = m_DataStructure.getDataRefAs(m_InputValues->FeatureReferenceMisorientationsArrayName); auto& avgReferenceMisorientation = m_DataStructure.getDataRefAs(m_InputValues->FeatureAvgMisorientationsArrayName); @@ -59,12 +63,10 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() return validateNumFeatResult; } - std::vector m_OrientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - - const size_t totalVoxels = featureIds.getNumberOfTuples(); + std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + const usize totalVoxels = featureIds.getNumberOfTuples(); - // Get the total features from the appropriate source.. - size_t totalFeatures = 0; + usize totalFeatures = 0; if(featureAttrMatPtr != nullptr) { totalFeatures = featureAttrMatPtr->getNumberOfTuples(); @@ -78,98 +80,130 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() return MakeErrorResult(-34900, "Total features was zero. The filter cannot proceed. Check either the feature attribute matrix or the average quaternions for proper size"); } - // Create local storage for teh centers and center distances - std::vector m_Centers(totalFeatures, 0); - std::vector m_CenterDistances(totalFeatures, 0.0f); + // Cache crystal structures locally (ensemble-level, tiny) + const usize numXtalEntries = crystalStructures.getNumberOfTuples(); + std::vector localCrystalStructures(numXtalEntries); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(localCrystalStructures.data(), numXtalEntries)); - // If the user selected "Misorientation from Feature Centers" + // Cache avgQuats locally (feature-level) — used in mode 0 + std::vector localAvgQuats; + if(m_InputValues->ReferenceOrientation == 0 && avgQuatsPtr != nullptr) + { + localAvgQuats.resize(totalFeatures * 4); + avgQuatsPtr->getDataStoreRef().copyIntoBuffer(0, nonstd::span(localAvgQuats.data(), totalFeatures * 4)); + } + + std::vector centerVoxels(totalFeatures, 0); + std::vector centerDistances(totalFeatures, 0.0f); + std::vector centerQuats; + + const auto& featureIdsStore = featureIds.getDataStoreRef(); + const auto& phasesStore = cellPhases.getDataStoreRef(); + const auto& quatsStore = quats.getDataStoreRef(); + auto& misoStore = featureReferenceMisorientations.getDataStoreRef(); + + // Mode 1: find center voxels using chunked I/O if(m_InputValues->ReferenceOrientation == 1) { - const auto& m_GBEuclideanDistances = m_DataStructure.getDataRefAs(m_InputValues->GBEuclideanDistancesArrayPath); - for(size_t voxelIdx = 0; voxelIdx < totalVoxels; voxelIdx++) + const auto& gbDistStore = m_DataStructure.getDataRefAs(m_InputValues->GBEuclideanDistancesArrayPath).getDataStoreRef(); + auto fidBuf = std::make_unique(k_ChunkTuples); + auto distBuf = std::make_unique(k_ChunkTuples); + + for(usize offset = 0; offset < totalVoxels; offset += k_ChunkTuples) { if(m_ShouldCancel) { return {}; } - - int32_t featureId = featureIds[voxelIdx]; - float32 distance = m_GBEuclideanDistances[voxelIdx]; - if(distance >= m_CenterDistances[featureId]) + const usize count = std::min(k_ChunkTuples, totalVoxels - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(fidBuf.get(), count)); + gbDistStore.copyIntoBuffer(offset, nonstd::span(distBuf.get(), count)); + for(usize i = 0; i < count; i++) { - m_CenterDistances[featureId] = distance; // Save the GB Distance value - m_Centers[featureId] = voxelIdx; // Save the voxel index for that value + const int32 featureId = fidBuf[i]; + if(featureId > 0 && distBuf[i] >= centerDistances[featureId]) + { + centerDistances[featureId] = distBuf[i]; + centerVoxels[featureId] = offset + i; + } } } const auto& euclideanCellCenters = m_DataStructure.getDataAs(m_InputValues->FeatureEuclideanCentersPath)->getIDataStoreAs>(); - - for(size_t i = 1; i < totalFeatures; i++) + for(usize i = 1; i < totalFeatures; i++) { - usize voxelIdx = m_Centers[i]; - auto cellCenter = imageGeom.getCoordsf(voxelIdx); + auto cellCenter = imageGeom.getCoordsf(centerVoxels[i]); euclideanCellCenters->setTuple(i, cellCenter.data()); } + + centerQuats.resize(totalFeatures * 4, 0.0f); + for(usize i = 1; i < totalFeatures; i++) + { + float32 qBuf[4] = {}; + quatsStore.copyIntoBuffer(centerVoxels[i] * 4, nonstd::span(qBuf, 4)); + centerQuats[i * 4 + 0] = qBuf[0]; + centerQuats[i * 4 + 1] = qBuf[1]; + centerQuats[i * 4 + 2] = qBuf[2]; + centerQuats[i * 4 + 3] = qBuf[3]; + } } - std::vector avgMisorientationSums(totalFeatures, 0.0F); - std::vector avgMisorientationCounts(totalFeatures, 0.0F); + std::vector avgMisorientationSums(totalFeatures, 0.0f); + std::vector avgMisorientationCounts(totalFeatures, 0.0f); + featureReferenceMisorientations.fill(0.0f); + + auto featureIdBuf = std::make_unique(k_ChunkTuples); + auto phasesBuf = std::make_unique(k_ChunkTuples); + auto quatsBuf = std::make_unique(k_ChunkTuples * 4); + auto misoBuf = std::make_unique(k_ChunkTuples); - featureReferenceMisorientations.fill(0.0f); // Fill all values with Zeros. - for(int64_t voxelIdx = 0; voxelIdx < totalVoxels; voxelIdx++) + for(usize offset = 0; offset < totalVoxels; offset += k_ChunkTuples) { if(m_ShouldCancel) { return {}; } + const usize count = std::min(k_ChunkTuples, totalVoxels - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), count)); + phasesStore.copyIntoBuffer(offset, nonstd::span(phasesBuf.get(), count)); + quatsStore.copyIntoBuffer(offset * 4, nonstd::span(quatsBuf.get(), count * 4)); + std::fill_n(misoBuf.get(), count, 0.0f); - if(featureIds[voxelIdx] > 0 && cellPhases[voxelIdx] > 0) + for(usize i = 0; i < count; i++) { - // Get the orientation of the current voxel - ebsdlib::QuatD q1(quats[voxelIdx * 4 + 0], quats[voxelIdx * 4 + 1], quats[voxelIdx * 4 + 2], quats[voxelIdx * 4 + 3]); - ebsdlib::QuatD q2; // Get this ready to use. It gets filled depending on the kind of reference orientation the user selected - if(m_InputValues->ReferenceOrientation == 0) // Use Average Quaternions - { - const auto featureId = static_cast(featureIds[voxelIdx]); - q2 = ebsdlib::QuatD(avgQuatsPtr->getValue(featureId * 4), avgQuatsPtr->getValue(featureId * 4 + 1), avgQuatsPtr->getValue(featureId * 4 + 2), avgQuatsPtr->getValue(featureId * 4 + 3)); - } - else if(m_InputValues->ReferenceOrientation == 1) // Use the voxel's orientation that is the farthest from the grain boundary + const int32 featureId = featureIdBuf[i]; + const int32 phase = phasesBuf[i]; + if(featureId > 0 && phase > 0) { - auto featureId = static_cast(featureIds[voxelIdx]); - size_t centerVoxelIdx = m_Centers[featureId]; - q2 = ebsdlib::QuatD(quats[centerVoxelIdx * 4 + 0], quats[centerVoxelIdx * 4 + 1], quats[centerVoxelIdx * 4 + 2], quats[centerVoxelIdx * 4 + 3]); + const usize qi = i * 4; + ebsdlib::QuatD q1(quatsBuf[qi], quatsBuf[qi + 1], quatsBuf[qi + 2], quatsBuf[qi + 3]); + ebsdlib::QuatD q2; + if(m_InputValues->ReferenceOrientation == 0) + { + const usize fi = static_cast(featureId) * 4; + q2 = ebsdlib::QuatD(localAvgQuats[fi], localAvgQuats[fi + 1], localAvgQuats[fi + 2], localAvgQuats[fi + 3]); + } + else if(m_InputValues->ReferenceOrientation == 1) + { + const usize fi = static_cast(featureId) * 4; + q2 = ebsdlib::QuatD(centerQuats[fi], centerQuats[fi + 1], centerQuats[fi + 2], centerQuats[fi + 3]); + } + + const uint32 laueClass = localCrystalStructures[phase]; + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass]->calculateMisorientation(q1, q2); + const float32 misoValue = static_cast(Constants::k_RadToDegD * axisAngle[3]); + misoBuf[i] = misoValue; + avgMisorientationCounts[featureId]++; + avgMisorientationSums[featureId] += misoValue; } - - uint32 laueClass1 = crystalStructures[cellPhases[voxelIdx]]; - ebsdlib::AxisAngleDType axisAngle = m_OrientationOps[laueClass1]->calculateMisorientation(q1, q2); - - // Extract the misorientation, convert it to degrees, and store if for this voxel - featureReferenceMisorientations[voxelIdx] = static_cast(Constants::k_RadToDegD * axisAngle[3]); // convert to degrees - - // Update our temp storage vectors that will eventually compute the final `average reference misorientation` - int32_t idx = featureIds[voxelIdx]; - avgMisorientationCounts[idx]++; - avgMisorientationSums[idx] = avgMisorientationSums[idx] + featureReferenceMisorientations[voxelIdx]; } + misoStore.copyFromBuffer(offset, nonstd::span(misoBuf.get(), count)); } - // Update the avgReferenceMisorientation output array avgReferenceMisorientation[0] = 0.0f; - for(size_t featureIdx = 1; featureIdx < totalFeatures; featureIdx++) + for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) { - if(m_ShouldCancel) - { - return {}; - } - - if(avgMisorientationCounts[featureIdx] == 0.0f) - { - avgReferenceMisorientation[featureIdx] = 0.0f; - } - else - { - avgReferenceMisorientation[featureIdx] = avgMisorientationSums[featureIdx] / avgMisorientationCounts[featureIdx]; - } + avgReferenceMisorientation[featureIdx] = (avgMisorientationCounts[featureIdx] == 0.0f) ? 0.0f : avgMisorientationSums[featureIdx] / avgMisorientationCounts[featureIdx]; } return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp index 11a1b3a258..83848bbc70 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp @@ -29,11 +29,11 @@ class CalculateGBCDImpl { usize m_TriangleChunkStartIndex; usize m_NumBinPerTriangle; - Int32Array& m_LabelsArray; - Float64Array& m_NormalsArray; - Int32Array& m_PhasesArray; - Float32Array& m_EulersArray; - UInt32Array& m_CrystalStructuresArray; + const int32* m_Labels; + const float64* m_Normals; + const int32* m_PhasesCache; + const float32* m_EulersCache; + const uint32* m_CrystalStructuresCache; SizeGBCD& m_SizeGBCD; LaueOpsContainerType m_OrientationOps; @@ -42,14 +42,15 @@ class CalculateGBCDImpl CalculateGBCDImpl() = delete; CalculateGBCDImpl(const CalculateGBCDImpl&) = default; - CalculateGBCDImpl(usize i, usize numMisoReps, Int32Array& labels, Float64Array& normals, Float32Array& eulers, Int32Array& phases, UInt32Array& crystalStructures, SizeGBCD& sizeGBCD) + CalculateGBCDImpl(usize i, usize numMisoReps, const int32* labels, const float64* normals, const float32* eulersCache, const int32* phasesCache, const uint32* crystalStructuresCache, + SizeGBCD& sizeGBCD) : m_TriangleChunkStartIndex(i) , m_NumBinPerTriangle(numMisoReps) - , m_LabelsArray(labels) - , m_NormalsArray(normals) - , m_PhasesArray(phases) - , m_EulersArray(eulers) - , m_CrystalStructuresArray(crystalStructures) + , m_Labels(labels) + , m_Normals(normals) + , m_PhasesCache(phasesCache) + , m_EulersCache(eulersCache) + , m_CrystalStructuresCache(crystalStructuresCache) , m_SizeGBCD(sizeGBCD) { m_OrientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); @@ -65,12 +66,6 @@ class CalculateGBCDImpl std::vector& gbcdBins = m_SizeGBCD.m_GbcdBins; std::vector& hemiCheck = m_SizeGBCD.m_GbcdHemiCheck; // Definitely do NOT want a raw pointer to vector because that done in bits, not bytes. - Int32Array& labels = m_LabelsArray; - Float64Array& normals = m_NormalsArray; - Int32Array& phases = m_PhasesArray; - Float32Array& eulers = m_EulersArray; - UInt32Array& crystalStructures = m_CrystalStructuresArray; - int32 feature1 = 0, feature2 = 0; int32 inversion = 1; float32 g1ea[3] = {0.0f, 0.0f, 0.0f}; @@ -79,9 +74,9 @@ class CalculateGBCDImpl // float32 g1s[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}, g2s[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; // float32 sym1[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}, sym2[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; // float32 g2t[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}, dg[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - std::array eulerMis = {0.0f, 0.0f, 0.0f}; - ebsdlib::Matrix3X1 normal; // = {0.0f, 0.0f, 0.0f}; - ebsdlib::Matrix3X1 xstl1Norm1; // {0.0f, 0.0f, 0.0f}; + std::array eulerMis = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 normal; // = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 xstl1Norm1; // {0.0f, 0.0f, 0.0f}; float32 sqCoord[2] = {0.0f, 0.0f}, sqCoordInv[2] = {0.0f, 0.0f}; for(usize triangleIndex = start; triangleIndex < end; triangleIndex++) @@ -89,8 +84,8 @@ class CalculateGBCDImpl usize minGbcdBinIndex = (triangleIndex - m_TriangleChunkStartIndex) * m_NumBinPerTriangle; int32 symCounter = 0; - feature1 = labels[2 * triangleIndex]; - feature2 = labels[2 * triangleIndex + 1]; + feature1 = m_Labels[2 * triangleIndex]; + feature2 = m_Labels[2 * triangleIndex + 1]; if(feature1 < 0 || feature2 < 0) { @@ -98,13 +93,13 @@ class CalculateGBCDImpl } // Get the normal for the triangle - normal[0] = normals[3 * triangleIndex]; - normal[1] = normals[3 * triangleIndex + 1]; - normal[2] = normals[3 * triangleIndex + 2]; + normal[0] = m_Normals[3 * triangleIndex]; + normal[1] = m_Normals[3 * triangleIndex + 1]; + normal[2] = m_Normals[3 * triangleIndex + 2]; - if(phases[feature1] == phases[feature2] && phases[feature1] > 0) + if(m_PhasesCache[feature1] == m_PhasesCache[feature2] && m_PhasesCache[feature1] > 0) { - uint32 laueClass1 = crystalStructures[phases[feature1]]; + uint32 laueClass1 = m_CrystalStructuresCache[m_PhasesCache[feature1]]; for(int32 q = 0; q < 2; q++) { if(q == 1) @@ -118,19 +113,19 @@ class CalculateGBCDImpl } for(int32 m = 0; m < 3; m++) { - g1ea[m] = eulers[3 * feature1 + m]; - g2ea[m] = eulers[3 * feature2 + m]; + g1ea[m] = m_EulersCache[3 * feature1 + m]; + g2ea[m] = m_EulersCache[3 * feature2 + m]; } - ebsdlib::Matrix3X3 g1 = ebsdlib::EulerFType(g1ea).toOrientationMatrix().toGMatrix(); - ebsdlib::Matrix3X3 g2 = ebsdlib::EulerFType(g2ea).toOrientationMatrix().toGMatrix(); + ebsdlib::Matrix3X3 g1 = ebsdlib::EulerFType(g1ea).toOrientationMatrix().toGMatrix(); + ebsdlib::Matrix3X3 g2 = ebsdlib::EulerFType(g2ea).toOrientationMatrix().toGMatrix(); int32 nSym = m_OrientationOps[laueClass1]->getNumSymOps(); for(int32 j = 0; j < nSym; j++) { // rotate g1 by symOp - ebsdlib::Matrix3X3 sym1 = m_OrientationOps[laueClass1]->getMatSymOpF(j); - ebsdlib::Matrix3X3 g1s = sym1 * g1; + ebsdlib::Matrix3X3 sym1 = m_OrientationOps[laueClass1]->getMatSymOpF(j); + ebsdlib::Matrix3X3 g1s = sym1 * g1; // get the crystal directions along the triangle normals xstl1Norm1 = g1s * normal; // get coordinates in square projection of crystal normal parallel to boundary normal @@ -146,13 +141,13 @@ class CalculateGBCDImpl for(int32 k = 0; k < nSym; k++) { // calculate the symmetric misorienation - ebsdlib::Matrix3X3 sym2 = m_OrientationOps[laueClass1]->getMatSymOpF(k); + ebsdlib::Matrix3X3 sym2 = m_OrientationOps[laueClass1]->getMatSymOpF(k); // rotate g2 by symOp - ebsdlib::Matrix3X3 g2s = sym2 * g2; + ebsdlib::Matrix3X3 g2s = sym2 * g2; // transpose rotated g2 - ebsdlib::Matrix3X3 g2t = g2s.transpose(); + ebsdlib::Matrix3X3 g2t = g2s.transpose(); // calculate delta g - ebsdlib::Matrix3X3 dg = g1s * g2t; + ebsdlib::Matrix3X3 dg = g1s * g2t; // translate matrix to euler angles ebsdlib::OrientationMatrixFType om(dg); @@ -379,7 +374,19 @@ Result<> ComputeGBCD::operator()() auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayName); + // Cache feature-level arrays locally to eliminate per-element OOC overhead + const usize numEulerElements = eulerAngles.getSize(); + std::vector eulersCache(numEulerElements); + eulerAngles.getDataStoreRef().copyIntoBuffer(0, nonstd::span(eulersCache.data(), numEulerElements)); + + const usize numPhaseElements = phases.getSize(); + std::vector phasesCache(numPhaseElements); + phases.getDataStoreRef().copyIntoBuffer(0, nonstd::span(phasesCache.data(), numPhaseElements)); + + // Cache ensemble-level arrays (tiny) usize totalPhases = crystalStructures.getNumberOfTuples(); + std::vector crystalStructuresCache(totalPhases); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.data(), totalPhases)); usize totalFaces = faceLabels.getNumberOfTuples(); usize triangleChunkSize = 50000; @@ -394,11 +401,24 @@ Result<> ComputeGBCD::operator()() MessageHelper messageHelper(m_MessageHandler); // create an array to hold the total face area for each phase and initialize the array to 0.0 - std::vector totalFaceArea(totalPhases, 0.0); + std::vector totalFaceArea(totalPhases, 0.0); auto startTime = std::chrono::steady_clock::now(); messageHelper.sendMessage("1/2 Starting GBCD Calculation and Summation Phase"); ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); + // Pre-allocate chunk buffers for triangle-level arrays + const auto& labelsStore = faceLabels.getDataStoreRef(); + const auto& normalsStore = faceNormals.getDataStoreRef(); + const auto& areasStore = faceAreas.getDataStoreRef(); + std::vector labelsBuf(triangleChunkSize * 2); + std::vector normalsBuf(triangleChunkSize * 3); + std::vector areasBuf(triangleChunkSize); + + // Cache the full GBCD output locally for accumulation — bounded by + // totalPhases * totalGBCDBins (bin resolution, not cell count) + const usize gbcdTotalElements = gbcd.getSize(); + std::vector gbcdBuf(gbcdTotalElements, 0.0); + for(usize i = 0; i < totalFaces; i = i + triangleChunkSize) { if(getCancel()) @@ -413,27 +433,30 @@ Result<> ComputeGBCD::operator()() sizeGbcd.initializeBinsWithValue(-1); sizeGbcd.m_GbcdHemiCheck.assign(sizeGbcd.m_GbcdHemiCheck.size(), false); + // Chunk-read triangle arrays for this iteration + labelsStore.copyIntoBuffer(i * 2, nonstd::span(labelsBuf.data(), triangleChunkSize * 2)); + normalsStore.copyIntoBuffer(i * 3, nonstd::span(normalsBuf.data(), triangleChunkSize * 3)); + areasStore.copyIntoBuffer(i, nonstd::span(areasBuf.data(), triangleChunkSize)); + ParallelDataAlgorithm parallelTask; parallelTask.setRange(i, i + triangleChunkSize); - parallelTask.execute(CalculateGBCDImpl(i, k_NumMisoReps, faceLabels, faceNormals, eulerAngles, phases, crystalStructures, sizeGbcd)); + parallelTask.execute(CalculateGBCDImpl(i, k_NumMisoReps, labelsBuf.data() - static_cast(i * 2), normalsBuf.data() - static_cast(i * 3), eulersCache.data(), + phasesCache.data(), crystalStructuresCache.data(), sizeGbcd)); if(getCancel()) { return {}; } - int32 phase = 0; - int32 feature = 0; - double area = 0.0; for(usize j = 0; j < triangleChunkSize; j++) { - area = faceAreas[i + j]; - feature = faceLabels[2 * (i + j)]; + float64 area = areasBuf[j]; + int32 feature = labelsBuf[2 * j]; if(feature < 0) { continue; } - phase = phases[feature]; + int32 phase = phasesCache[feature]; for(usize k = 0; k < k_NumMisoReps; k++) { usize gbcdBinIdx = (j * k_NumMisoReps) + k; @@ -446,12 +469,11 @@ Result<> ComputeGBCD::operator()() hemisphere = 1; } usize gbcdIdx = (phase * totalGBCDBins) + (2 * sizeGbcd.m_GbcdBins[gbcdBinIdx] + hemisphere); - gbcd[gbcdIdx] += area; + gbcdBuf[gbcdIdx] += area; totalFaceArea[phase] += area; } } } - throttledMessenger.sendThrottledMessage([&]() { auto currentTime = throttledMessenger.getLastTime(); const usize k_LastTriangleIndex = i + triangleChunkSize; @@ -464,15 +486,17 @@ Result<> ComputeGBCD::operator()() messageHelper.sendMessage("2/2 Starting GBCD Normalization Phase"); - for(int32 i = 0; i < totalPhases; i++) + // Normalize GBCD in the local buffer, then write back to the DataStore + for(usize i = 0; i < totalPhases; i++) { - const usize k_PhaseShift = i * totalGBCDBins; - const double k_MrdFactor = static_cast(totalGBCDBins) / totalFaceArea[i]; - for(int32 j = 0; j < totalGBCDBins; j++) + const usize k_PhaseShift = i * static_cast(totalGBCDBins); + const float64 k_MrdFactor = static_cast(totalGBCDBins) / totalFaceArea[i]; + for(usize j = 0; j < static_cast(totalGBCDBins); j++) { - gbcd[k_PhaseShift + j] *= k_MrdFactor; + gbcdBuf[k_PhaseShift + j] *= k_MrdFactor; } } + gbcd.getDataStoreRef().copyFromBuffer(0, nonstd::span(gbcdBuf.data(), gbcdTotalElements)); return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp index 139da97040..b3bfe85728 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp @@ -70,24 +70,23 @@ class TrianglesSelector #else std::vector& selectedTriangles, #endif - std::vector& triIncluded, float64 misResolution, int32 phaseOfInterest, const Matrix3dR& gFixedT, const UInt32Array& crystalStructures, const Float32Array& euler, - const Int32Array& phases, const Int32Array& faceLabels, const Float64Array& faceNormals, const Float64Array& faceAreas) + float64 misResolution, int32 phaseOfInterest, const Matrix3dR& gFixedT, uint32 crystalStruct, const float32* eulerCache, const int32* phasesCache, const Int32Array& faceLabels, + const Float64Array& faceNormals, const Float64Array& faceAreas) : m_ExcludeTripleLines(excludeTripleLines) , m_Triangles(triangles) , m_NodeTypes(nodeTypes) , m_SelectedTriangles(selectedTriangles) - , m_TriIncluded(triIncluded) , m_MisResolution(misResolution) , m_PhaseOfInterest(phaseOfInterest) , m_GFixedT(gFixedT) - , m_Euler(euler) - , m_Phases(phases) + , m_EulerCache(eulerCache) + , m_PhasesCache(phasesCache) , m_FaceLabels(faceLabels) , m_FaceNormals(faceNormals) , m_FaceAreas(faceAreas) { m_OrientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - m_Crystal = crystalStructures[phaseOfInterest]; + m_Crystal = crystalStruct; m_NSym = m_OrientationOps[m_Crystal]->getNumSymOps(); } @@ -121,11 +120,11 @@ class TrianglesSelector { continue; } - if(m_Phases[feature1] != m_Phases[feature2]) + if(m_PhasesCache[feature1] != m_PhasesCache[feature2]) { continue; } - if(m_Phases[feature1] != m_PhaseOfInterest || m_Phases[feature2] != m_PhaseOfInterest) + if(m_PhasesCache[feature1] != m_PhaseOfInterest || m_PhasesCache[feature2] != m_PhaseOfInterest) { continue; } @@ -138,16 +137,14 @@ class TrianglesSelector } } - m_TriIncluded[triIdx] = 1; - normalLab[0] = (m_FaceNormals[3 * triIdx]); normalLab[1] = (m_FaceNormals[3 * triIdx + 1]); normalLab[2] = (m_FaceNormals[3 * triIdx + 2]); for(int whichEa = 0; whichEa < 3; whichEa++) { - g1ea[whichEa] = m_Euler[3 * feature1 + whichEa]; - g2ea[whichEa] = m_Euler[3 * feature2 + whichEa]; + g1ea[whichEa] = m_EulerCache[3 * feature1 + whichEa]; + g2ea[whichEa] = m_EulerCache[3 * feature2 + whichEa]; } auto oMatrix1 = ebsdlib::EulerDType(g1ea[0], g1ea[1], g1ea[2]).toOrientationMatrix(); @@ -219,7 +216,6 @@ class TrianglesSelector #else std::vector& m_SelectedTriangles; #endif - std::vector& m_TriIncluded; float64 m_MisResolution; int32 m_PhaseOfInterest; const Matrix3dR& m_GFixedT; @@ -228,8 +224,8 @@ class TrianglesSelector uint32 m_Crystal; int32 m_NSym; - const Float32Array& m_Euler; - const Int32Array& m_Phases; + const float32* m_EulerCache; + const int32* m_PhasesCache; const Int32Array& m_FaceLabels; const Float64Array& m_FaceNormals; const Float64Array& m_FaceAreas; @@ -399,11 +395,30 @@ Result<> ComputeGBCDMetricBased::operator()() auto& triangleGeom = m_DataStructure.getDataRefAs(m_InputValues->TriangleGeometryPath); const IGeometry::SharedFaceList& triangles = triangleGeom.getFacesRef(); + // Cache feature-level arrays locally to eliminate per-element OOC overhead + const usize numEulerElements = eulerAngles.getSize(); + std::vector eulerCache(numEulerElements); + eulerAngles.getDataStoreRef().copyIntoBuffer(0, nonstd::span(eulerCache.data(), numEulerElements)); + + const usize numPhaseElements = phases.getSize(); + std::vector phasesCache(numPhaseElements); + phases.getDataStoreRef().copyIntoBuffer(0, nonstd::span(phasesCache.data(), numPhaseElements)); + + // Cache ensemble-level arrays (tiny) + const usize numCrystalStructures = crystalStructures.getSize(); + std::vector crystalStructuresCache(numCrystalStructures); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.data(), numCrystalStructures)); + + // Cache feature-face labels (feature-level) + const usize numFeatureFaceElements = featureFaceLabels.getSize(); + std::vector featureFaceLabelsCache(numFeatureFaceElements); + featureFaceLabels.getDataStoreRef().copyIntoBuffer(0, nonstd::span(featureFaceLabelsCache.data(), numFeatureFaceElements)); + // ------------------- before computing the distribution, we must find normalization factors ----- float64 ballVolume = k_BallVolumesM3M[m_InputValues->ChosenLimitDists]; { std::vector ops = ebsdlib::LaueOps::GetAllOrientationOps(); - auto crystalStruct = static_cast(crystalStructures[m_InputValues->PhaseOfInterest]); + auto crystalStruct = static_cast(crystalStructuresCache[m_InputValues->PhaseOfInterest]); const int32 nSym = ops[crystalStruct]->getNumSymOps(); if(crystalStruct != 1) @@ -470,14 +485,16 @@ Result<> ComputeGBCDMetricBased::operator()() std::vector selectedTriangles(0); #endif - std::vector triIncluded(numMeshTriangles, 0); - usize triChunkSize = 50000; if(numMeshTriangles < triChunkSize) { triChunkSize = numMeshTriangles; } + // Accumulate totalFaceArea per-chunk by re-checking the geometric filter + // conditions instead of storing an O(n) triIncluded mask. + float64 totalFaceArea = 0.0; + for(usize i = 0; i < numMeshTriangles; i += triChunkSize) { if(getCancel()) @@ -486,16 +503,47 @@ Result<> ComputeGBCDMetricBased::operator()() } m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Step 1/2: Selecting Triangles with the Specified Misorientation ({}% completed)", static_cast(100.0 * static_cast(i) / static_cast(numMeshTriangles)))); - if(i + triChunkSize >= numMeshTriangles) + usize currentChunkSize = triChunkSize; + if(i + currentChunkSize >= numMeshTriangles) { - triChunkSize = numMeshTriangles - i; + currentChunkSize = numMeshTriangles - i; } ParallelDataAlgorithm dataAlg; - dataAlg.setRange(i, i + triChunkSize); + dataAlg.setRange(i, i + currentChunkSize); dataAlg.setParallelizationEnabled(true); - dataAlg.execute(GBCDMetricBased::TrianglesSelector(m_InputValues->ExcludeTripleLines, triangles, nodeTypes, selectedTriangles, triIncluded, misResolution, m_InputValues->PhaseOfInterest, gFixedT, - crystalStructures, eulerAngles, phases, faceLabels, faceNormals, faceAreas)); + dataAlg.execute(GBCDMetricBased::TrianglesSelector(m_InputValues->ExcludeTripleLines, triangles, nodeTypes, selectedTriangles, misResolution, m_InputValues->PhaseOfInterest, gFixedT, + crystalStructuresCache[m_InputValues->PhaseOfInterest], eulerCache.data(), phasesCache.data(), faceLabels, faceNormals, faceAreas)); + + // Chunk-read triangle arrays for totalFaceArea accumulation + { + std::vector labelsBuf(currentChunkSize * 2); + std::vector areasBuf(currentChunkSize); + faceLabels.getDataStoreRef().copyIntoBuffer(i * 2, nonstd::span(labelsBuf.data(), currentChunkSize * 2)); + faceAreas.getDataStoreRef().copyIntoBuffer(i, nonstd::span(areasBuf.data(), currentChunkSize)); + + for(usize j = 0; j < currentChunkSize; j++) + { + const int32 feature1 = labelsBuf[2 * j]; + const int32 feature2 = labelsBuf[2 * j + 1]; + if(feature1 < 1 || feature2 < 1) + { + continue; + } + if(phasesCache[feature1] != m_InputValues->PhaseOfInterest || phasesCache[feature2] != m_InputValues->PhaseOfInterest) + { + continue; + } + if(m_InputValues->ExcludeTripleLines) + { + if(nodeTypes[triangles[(i + j) * 3]] != 2 || nodeTypes[triangles[(i + j) * 3 + 1]] != 2 || nodeTypes[triangles[(i + j) * 3 + 2]] != 2) + { + continue; + } + } + totalFaceArea += areasBuf[j]; + } + } } // ------------------------ find the number of distinct boundaries ------------------------------ @@ -509,18 +557,18 @@ Result<> ComputeGBCDMetricBased::operator()() return {}; } - const int32 feature1 = featureFaceLabels[2 * featureFaceIdx]; - const int32 feature2 = featureFaceLabels[2 * featureFaceIdx + 1]; + const int32 feature1 = featureFaceLabelsCache[2 * featureFaceIdx]; + const int32 feature2 = featureFaceLabelsCache[2 * featureFaceIdx + 1]; if(feature1 < 1 || feature2 < 1) { continue; } - if(phases[feature1] != phases[feature2]) + if(phasesCache[feature1] != phasesCache[feature2]) { continue; } - if(phases[feature1] != m_InputValues->PhaseOfInterest || phases[feature2] != m_InputValues->PhaseOfInterest) + if(phasesCache[feature1] != m_InputValues->PhaseOfInterest || phasesCache[feature2] != m_InputValues->PhaseOfInterest) { continue; } @@ -528,13 +576,6 @@ Result<> ComputeGBCDMetricBased::operator()() numDistinctGBs++; } - // ----------------- determining distribution values at the sampling points (and their errors) --- - float64 totalFaceArea = 0.0; - for(usize triIdx = 0; triIdx < numMeshTriangles; triIdx++) - { - totalFaceArea += faceAreas[triIdx] * static_cast(triIncluded.at(triIdx)); - } - std::vector distributionValues(samplePtsX.size(), 0.0); std::vector errorValues(samplePtsX.size(), 0.0); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp similarity index 77% rename from src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.cpp rename to src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp index 179c3b4028..00622341a9 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp @@ -1,4 +1,4 @@ -#include "ComputeGBCDPoleFigure.hpp" +#include "ComputeGBCDPoleFigureDirect.hpp" #include "simplnx/Common/Array.hpp" #include "simplnx/Common/Constants.hpp" @@ -18,27 +18,27 @@ namespace class ComputeGBCDPoleFigureImpl { private: - Float64Array& m_PoleFigure; + float64* m_PoleFigure; std::array m_Dimensions; ebsdlib::LaueOps::Pointer m_OrientOps; const std::vector& m_GbcdDeltas; const std::vector& m_GbcdLimits; const std::vector& m_GbcdSizes; - const Float64Array& m_Gbcd; + const float64* m_Gbcd; int32 m_PhaseOfInterest = 0; const std::vector& m_MisorientationRotation; public: - ComputeGBCDPoleFigureImpl(Float64Array& poleFigureArray, const std::array& dimensions, const ebsdlib::LaueOps::Pointer& orientOps, const std::vector& gbcdDeltasArray, - const std::vector& gbcdLimitsArray, const std::vector& gbcdSizesArray, const Float64Array& gbcd, int32 phaseOfInterest, + ComputeGBCDPoleFigureImpl(float64* poleFigurePtr, const std::array& dimensions, const ebsdlib::LaueOps::Pointer& orientOps, const std::vector& gbcdDeltasArray, + const std::vector& gbcdLimitsArray, const std::vector& gbcdSizesArray, const float64* gbcdPtr, int32 phaseOfInterest, const std::vector& misorientationRotation) - : m_PoleFigure(poleFigureArray) + : m_PoleFigure(poleFigurePtr) , m_Dimensions(dimensions) , m_OrientOps(orientOps) , m_GbcdDeltas(gbcdDeltasArray) , m_GbcdLimits(gbcdLimitsArray) , m_GbcdSizes(gbcdSizesArray) - , m_Gbcd(gbcd) + , m_Gbcd(gbcdPtr) , m_PhaseOfInterest(phaseOfInterest) , m_MisorientationRotation(misorientationRotation) { @@ -47,27 +47,27 @@ class ComputeGBCDPoleFigureImpl void generate(usize xStart, usize xEnd, usize yStart, usize yEnd) const { - ebsdlib::Matrix3X1 vec = {0.0f, 0.0f, 0.0f}; - ebsdlib::Matrix3X1 vec2 = {0.0f, 0.0f, 0.0f}; - ebsdlib::Matrix3X1 rotNormal = {0.0f, 0.0f, 0.0f}; - ebsdlib::Matrix3X1 rotNormal2 = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 vec = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 vec2 = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 rotNormal = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 rotNormal2 = {0.0f, 0.0f, 0.0f}; std::array sqCoord = {0.0f, 0.0f}; // float32 dg[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; // float32 dgt[3][3] = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - ebsdlib::Matrix3X3 dg1; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - ebsdlib::Matrix3X3 dg2; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - ebsdlib::Matrix3X3 sym1; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - ebsdlib::Matrix3X3 sym2; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - ebsdlib::Matrix3X3 sym2t; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; - // Matrix3X1 misEuler1 = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X3 dg1; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; + ebsdlib::Matrix3X3 dg2; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; + ebsdlib::Matrix3X3 sym1; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; + ebsdlib::Matrix3X3 sym2; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; + ebsdlib::Matrix3X3 sym2t; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; + // Matrix3X1 misEuler1 = {0.0f, 0.0f, 0.0f}; float32 misAngle = m_MisorientationRotation[0] * nx::core::Constants::k_PiOver180F; nx::core::FloatVec3 normAxis = {m_MisorientationRotation[1], m_MisorientationRotation[2], m_MisorientationRotation[3]}; normAxis = normAxis.normalize(); // convert axis angle to matrix representation of misorientation - ebsdlib::Matrix3X3 dg = ebsdlib::AxisAngleFType(normAxis[0], normAxis[1], normAxis[2], misAngle).toOrientationMatrix().toGMatrix(); + ebsdlib::Matrix3X3 dg = ebsdlib::AxisAngleFType(normAxis[0], normAxis[1], normAxis[2], misAngle).toOrientationMatrix().toGMatrix(); // take inverse of misorientation variable to use for switching symmetry - ebsdlib::Matrix3X3 dgt = dg.transpose(); + ebsdlib::Matrix3X3 dgt = dg.transpose(); // get number of symmetry operators int32 nSym = m_OrientOps->getNumSymOps(); @@ -233,8 +233,8 @@ class ComputeGBCDPoleFigureImpl } // namespace // ----------------------------------------------------------------------------- -ComputeGBCDPoleFigure::ComputeGBCDPoleFigure(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - ComputeGBCDPoleFigureInputValues* inputValues) +ComputeGBCDPoleFigureDirect::ComputeGBCDPoleFigureDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeGBCDPoleFigureInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -243,21 +243,37 @@ ComputeGBCDPoleFigure::ComputeGBCDPoleFigure(DataStructure& dataStructure, const } // ----------------------------------------------------------------------------- -ComputeGBCDPoleFigure::~ComputeGBCDPoleFigure() noexcept = default; +ComputeGBCDPoleFigureDirect::~ComputeGBCDPoleFigureDirect() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ComputeGBCDPoleFigure::getCancel() +const std::atomic_bool& ComputeGBCDPoleFigureDirect::getCancel() { return m_ShouldCancel; } // ----------------------------------------------------------------------------- -Result<> ComputeGBCDPoleFigure::operator()() +Result<> ComputeGBCDPoleFigureDirect::operator()() { auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); - auto crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); DataPath cellIntensityArrayPath = m_InputValues->ImageGeometryPath.createChildPath(m_InputValues->CellAttributeMatrixName).createChildPath(m_InputValues->CellIntensityArrayName); - auto poleFigure = m_DataStructure.getDataRefAs(cellIntensityArrayPath); + auto& poleFigure = m_DataStructure.getDataRefAs(cellIntensityArrayPath); + + // Cache entire GBCD array locally — this is the in-core (Direct) path + // where the full array fits in RAM. + const usize gbcdTotalElements = gbcd.getSize(); + auto gbcdCache = std::make_unique(gbcdTotalElements); + gbcd.getDataStoreRef().copyIntoBuffer(0, nonstd::span(gbcdCache.get(), gbcdTotalElements)); + + // Cache crystal structures (ensemble-level, tiny) + const usize numCrystalStructures = crystalStructures.getSize(); + auto crystalStructuresCache = std::make_unique(numCrystalStructures); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.get(), numCrystalStructures)); + + // Cache pole figure output locally (300x300 = 90,000 elements, tiny) + const usize poleFigureSize = poleFigure.getSize(); + auto poleFigureCache = std::make_unique(poleFigureSize); + std::fill(poleFigureCache.get(), poleFigureCache.get() + poleFigureSize, 0.0); std::vector gbcdDeltas(5, 0); std::vector gbcdLimits(10, 0); @@ -309,7 +325,7 @@ Result<> ComputeGBCDPoleFigure::operator()() gbcdDeltas[4] = (gbcdLimits[9] - gbcdLimits[4]) / static_cast(gbcdSizes[4]); // Get our LaueOps pointer for the selected crystal structure - ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructures[m_InputValues->PhaseOfInterest]]; + ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructuresCache[m_InputValues->PhaseOfInterest]]; int32 xPoints = m_InputValues->OutputImageDimension; int32 yPoints = m_InputValues->OutputImageDimension; @@ -320,15 +336,15 @@ Result<> ComputeGBCDPoleFigure::operator()() m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Generating Intensity Plot for phase {}", m_InputValues->PhaseOfInterest)}); - typename IParallelAlgorithm::AlgorithmArrays algArrays; - algArrays.push_back(&poleFigure); - algArrays.push_back(&gbcd); - + // Use cached raw pointers — no OOC access in hot loop, parallelization is safe ParallelData2DAlgorithm dataAlg; dataAlg.setRange(0, xPoints, 0, yPoints); - dataAlg.requireArraysInMemory(algArrays); - dataAlg.execute(ComputeGBCDPoleFigureImpl(poleFigure, {xPoints, yPoints}, orientOps, gbcdDeltas, gbcdLimits, gbcdSizes, gbcd, m_InputValues->PhaseOfInterest, m_InputValues->MisorientationRotation)); + dataAlg.execute(ComputeGBCDPoleFigureImpl(poleFigureCache.get(), {xPoints, yPoints}, orientOps, gbcdDeltas, gbcdLimits, gbcdSizes, gbcdCache.get(), m_InputValues->PhaseOfInterest, + m_InputValues->MisorientationRotation)); + + // Write pole figure results back to the OOC store + poleFigure.getDataStoreRef().copyFromBuffer(0, nonstd::span(poleFigureCache.get(), poleFigureSize)); return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp similarity index 60% rename from src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.hpp rename to src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp index b508aab39f..a2847945b3 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp @@ -25,16 +25,16 @@ struct ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureInputValues /** * @class */ -class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigure +class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureDirect { public: - ComputeGBCDPoleFigure(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeGBCDPoleFigureInputValues* inputValues); - ~ComputeGBCDPoleFigure() noexcept; + ComputeGBCDPoleFigureDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeGBCDPoleFigureInputValues* inputValues); + ~ComputeGBCDPoleFigureDirect() noexcept; - ComputeGBCDPoleFigure(const ComputeGBCDPoleFigure&) = delete; - ComputeGBCDPoleFigure(ComputeGBCDPoleFigure&&) noexcept = delete; - ComputeGBCDPoleFigure& operator=(const ComputeGBCDPoleFigure&) = delete; - ComputeGBCDPoleFigure& operator=(ComputeGBCDPoleFigure&&) noexcept = delete; + ComputeGBCDPoleFigureDirect(const ComputeGBCDPoleFigureDirect&) = delete; + ComputeGBCDPoleFigureDirect(ComputeGBCDPoleFigureDirect&&) noexcept = delete; + ComputeGBCDPoleFigureDirect& operator=(const ComputeGBCDPoleFigureDirect&) = delete; + ComputeGBCDPoleFigureDirect& operator=(ComputeGBCDPoleFigureDirect&&) noexcept = delete; Result<> operator()(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp new file mode 100644 index 0000000000..e4226b2cc5 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp @@ -0,0 +1,309 @@ +#include "ComputeGBCDPoleFigureScanline.hpp" + +#include "ComputeGBCDPoleFigureDirect.hpp" // for ComputeGBCDPoleFigureInputValues + +#include "simplnx/Common/Array.hpp" +#include "simplnx/Common/Constants.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/ParallelData2DAlgorithm.hpp" + +#include +#include +#include + +namespace fs = std::filesystem; + +using namespace nx::core; + +namespace +{ +class ComputeGBCDPoleFigureImpl +{ +private: + float64* m_PoleFigure; + std::array m_Dimensions; + ebsdlib::LaueOps::Pointer m_OrientOps; + const std::vector& m_GbcdDeltas; + const std::vector& m_GbcdLimits; + const std::vector& m_GbcdSizes; + const float64* m_Gbcd; + int32 m_PhaseOfInterest = 0; + const std::vector& m_MisorientationRotation; + +public: + ComputeGBCDPoleFigureImpl(float64* poleFigurePtr, const std::array& dimensions, const ebsdlib::LaueOps::Pointer& orientOps, const std::vector& gbcdDeltasArray, + const std::vector& gbcdLimitsArray, const std::vector& gbcdSizesArray, const float64* gbcdPtr, int32 phaseOfInterest, + const std::vector& misorientationRotation) + : m_PoleFigure(poleFigurePtr) + , m_Dimensions(dimensions) + , m_OrientOps(orientOps) + , m_GbcdDeltas(gbcdDeltasArray) + , m_GbcdLimits(gbcdLimitsArray) + , m_GbcdSizes(gbcdSizesArray) + , m_Gbcd(gbcdPtr) + , m_PhaseOfInterest(phaseOfInterest) + , m_MisorientationRotation(misorientationRotation) + { + } + ~ComputeGBCDPoleFigureImpl() = default; + + void generate(usize xStart, usize xEnd, usize yStart, usize yEnd) const + { + ebsdlib::Matrix3X1 vec = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 vec2 = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 rotNormal = {0.0f, 0.0f, 0.0f}; + ebsdlib::Matrix3X1 rotNormal2 = {0.0f, 0.0f, 0.0f}; + std::array sqCoord = {0.0f, 0.0f}; + ebsdlib::Matrix3X3 dg1; + ebsdlib::Matrix3X3 dg2; + ebsdlib::Matrix3X3 sym1; + ebsdlib::Matrix3X3 sym2; + ebsdlib::Matrix3X3 sym2t; + + float32 misAngle = m_MisorientationRotation[0] * nx::core::Constants::k_PiOver180F; + nx::core::FloatVec3 normAxis = {m_MisorientationRotation[1], m_MisorientationRotation[2], m_MisorientationRotation[3]}; + normAxis = normAxis.normalize(); + ebsdlib::Matrix3X3 dg = ebsdlib::AxisAngleFType(normAxis[0], normAxis[1], normAxis[2], misAngle).toOrientationMatrix().toGMatrix(); + ebsdlib::Matrix3X3 dgt = dg.transpose(); + + int32 nSym = m_OrientOps->getNumSymOps(); + + int32 xPoints = m_Dimensions[0]; + int32 yPoints = m_Dimensions[1]; + int32 xPointsHalf = xPoints / 2; + int32 yPointsHalf = yPoints / 2; + float32 xRes = 2.0f / float32(xPoints); + float32 yRes = 2.0f / float32(yPoints); + bool nhCheck = false; + int32 hemisphere = 0; + + int32 shift1 = m_GbcdSizes[0]; + int32 shift2 = m_GbcdSizes[0] * m_GbcdSizes[1]; + int32 shift3 = m_GbcdSizes[0] * m_GbcdSizes[1] * m_GbcdSizes[2]; + int32 shift4 = m_GbcdSizes[0] * m_GbcdSizes[1] * m_GbcdSizes[2] * m_GbcdSizes[3]; + + int64 totalGbcdBins = m_GbcdSizes[0] * m_GbcdSizes[1] * m_GbcdSizes[2] * m_GbcdSizes[3] * m_GbcdSizes[4] * 2; + + for(int32 k = yStart; k < yEnd; k++) + { + for(int32 l = xStart; l < xEnd; l++) + { + float32 x = static_cast(l - xPointsHalf) * xRes + (xRes / 2.0F); + float32 y = static_cast(k - yPointsHalf) * yRes + (yRes / 2.0F); + + if((x * x + y * y) <= 1.0) + { + float64 sum = 0.0; + int32 count = 0; + vec[2] = -((x * x + y * y) - 1) / ((x * x + y * y) + 1); + vec[0] = x * (1 + vec[2]); + vec[1] = y * (1 + vec[2]); + vec2 = dgt * vec; + + for(int32 i = 0; i < nSym; i++) + { + sym1 = m_OrientOps->getMatSymOpF(i); + for(int32 j = 0; j < nSym; j++) + { + sym2 = m_OrientOps->getMatSymOpF(j); + sym2t = sym2.transpose(); + dg1 = dg * sym2t; + dg2 = sym1 * dg1; + + ebsdlib::EulerFType misEuler1 = ebsdlib::OrientationMatrixFType(dg2).toEuler(); + if(misEuler1[0] < nx::core::Constants::k_PiOver2F && misEuler1[1] < nx::core::Constants::k_PiOver2F && misEuler1[2] < nx::core::Constants::k_PiOver2F) + { + misEuler1[1] = cosf(misEuler1[1]); + auto location1 = static_cast((misEuler1[0] - m_GbcdLimits[0]) / m_GbcdDeltas[0]); + auto location2 = static_cast((misEuler1[1] - m_GbcdLimits[1]) / m_GbcdDeltas[1]); + auto location3 = static_cast((misEuler1[2] - m_GbcdLimits[2]) / m_GbcdDeltas[2]); + rotNormal = sym1 * vec; + nhCheck = getSquareCoord(rotNormal.data(), sqCoord.data()); + auto location4 = static_cast((sqCoord[0] - m_GbcdLimits[3]) / m_GbcdDeltas[3]); + auto location5 = static_cast((sqCoord[1] - m_GbcdLimits[4]) / m_GbcdDeltas[4]); + if(location1 >= 0 && location2 >= 0 && location3 >= 0 && location4 >= 0 && location5 >= 0 && location1 < m_GbcdSizes[0] && location2 < m_GbcdSizes[1] && location3 < m_GbcdSizes[2] && + location4 < m_GbcdSizes[3] && location5 < m_GbcdSizes[4]) + { + hemisphere = 0; + if(!nhCheck) + { + hemisphere = 1; + } + // m_Gbcd points to the phase-of-interest slice, so index directly without phase offset + sum += m_Gbcd[2 * ((location5 * shift4) + (location4 * shift3) + (location3 * shift2) + (location2 * shift1) + location1) + hemisphere]; + count++; + } + } + + // again in second crystal reference frame + dg1 = dgt * sym2; + dg2 = sym1 * dg1; + misEuler1 = ebsdlib::OrientationMatrixFType(dg2).toEuler(); + if(misEuler1[0] < nx::core::Constants::k_PiOver2D && misEuler1[1] < nx::core::Constants::k_PiOver2F && misEuler1[2] < nx::core::Constants::k_PiOver2F) + { + misEuler1[1] = cosf(misEuler1[1]); + auto location1 = static_cast((misEuler1[0] - m_GbcdLimits[0]) / m_GbcdDeltas[0]); + auto location2 = static_cast((misEuler1[1] - m_GbcdLimits[1]) / m_GbcdDeltas[1]); + auto location3 = static_cast((misEuler1[2] - m_GbcdLimits[2]) / m_GbcdDeltas[2]); + rotNormal2 = sym1 * vec2; + nhCheck = getSquareCoord(rotNormal2.data(), sqCoord.data()); + auto location4 = static_cast((sqCoord[0] - m_GbcdLimits[3]) / m_GbcdDeltas[3]); + auto location5 = static_cast((sqCoord[1] - m_GbcdLimits[4]) / m_GbcdDeltas[4]); + if(location1 >= 0 && location2 >= 0 && location3 >= 0 && location4 >= 0 && location5 >= 0 && location1 < m_GbcdSizes[0] && location2 < m_GbcdSizes[1] && location3 < m_GbcdSizes[2] && + location4 < m_GbcdSizes[3] && location5 < m_GbcdSizes[4]) + { + hemisphere = 0; + if(!nhCheck) + { + hemisphere = 1; + } + sum += m_Gbcd[2 * ((location5 * shift4) + (location4 * shift3) + (location3 * shift2) + (location2 * shift1) + location1) + hemisphere]; + count++; + } + } + } + } + if(count > 0) + { + m_PoleFigure[(k * xPoints) + l] = sum / float32(count); + } + } + } + } + } + + void operator()(const Range2D& r) const + { + generate(r.minCol(), r.maxCol(), r.minRow(), r.maxRow()); + } + +private: + static bool getSquareCoord(float32* crystalNormal, float32* sqCoord) + { + bool nhCheck = false; + float32 adjust = 1.0; + if(crystalNormal[2] >= 0.0) + { + adjust = -1.0; + nhCheck = true; + } + if(fabsf(crystalNormal[0]) >= fabsf(crystalNormal[1])) + { + sqCoord[0] = (crystalNormal[0] / fabsf(crystalNormal[0])) * sqrtf(2.0f * 1.0f * (1.0f + (crystalNormal[2] * adjust))) * (nx::core::Constants::k_SqrtPiF / 2.0f); + sqCoord[1] = (crystalNormal[0] / fabsf(crystalNormal[0])) * sqrtf(2.0f * 1.0f * (1.0f + (crystalNormal[2] * adjust))) * + ((2.0f / nx::core::Constants::k_SqrtPiF) * atanf(crystalNormal[1] / crystalNormal[0])); + } + else + { + sqCoord[0] = (crystalNormal[1] / fabsf(crystalNormal[1])) * sqrtf(2.0f * 1.0f * (1.0f + (crystalNormal[2] * adjust))) * + ((2.0f / nx::core::Constants::k_SqrtPiF) * atanf(crystalNormal[0] / crystalNormal[1])); + sqCoord[1] = (crystalNormal[1] / fabsf(crystalNormal[1])) * sqrtf(2.0f * 1.0f * (1.0f + (crystalNormal[2] * adjust))) * (nx::core::Constants::k_SqrtPiF / 2.0f); + } + return nhCheck; + } +}; + +} // namespace + +// ----------------------------------------------------------------------------- +ComputeGBCDPoleFigureScanline::ComputeGBCDPoleFigureScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeGBCDPoleFigureInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeGBCDPoleFigureScanline::~ComputeGBCDPoleFigureScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +const std::atomic_bool& ComputeGBCDPoleFigureScanline::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> ComputeGBCDPoleFigureScanline::operator()() +{ + auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); + auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + DataPath cellIntensityArrayPath = m_InputValues->ImageGeometryPath.createChildPath(m_InputValues->CellAttributeMatrixName).createChildPath(m_InputValues->CellIntensityArrayName); + auto& poleFigure = m_DataStructure.getDataRefAs(cellIntensityArrayPath); + + // Cache crystal structures (ensemble-level, tiny) + const usize numCrystalStructures = crystalStructures.getSize(); + auto crystalStructuresCache = std::make_unique(numCrystalStructures); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.get(), numCrystalStructures)); + + // Cache pole figure output locally (300x300 = 90,000 elements, tiny) + const usize poleFigureSize = poleFigure.getSize(); + auto poleFigureCache = std::make_unique(poleFigureSize); + std::fill(poleFigureCache.get(), poleFigureCache.get() + poleFigureSize, 0.0); + + std::vector gbcdDeltas(5, 0); + std::vector gbcdLimits(10, 0); + std::vector gbcdSizes(5, 0); + + // Greg R. Ranges + gbcdLimits[0] = 0.0f; + gbcdLimits[1] = 0.0f; + gbcdLimits[2] = 0.0f; + gbcdLimits[3] = 0.0f; + gbcdLimits[4] = 0.0f; + gbcdLimits[5] = Constants::k_PiOver2D; + gbcdLimits[6] = 1.0f; + gbcdLimits[7] = Constants::k_PiOver2D; + gbcdLimits[8] = 1.0f; + gbcdLimits[9] = Constants::k_2PiD; + + // reset the 3rd and 4th dimensions using the square grid approach + gbcdLimits[3] = -sqrtf(Constants::k_PiOver2D); + gbcdLimits[4] = -sqrtf(Constants::k_PiOver2D); + gbcdLimits[8] = sqrtf(Constants::k_PiOver2D); + gbcdLimits[9] = sqrtf(Constants::k_PiOver2D); + + // get num components of GBCD + ShapeType cDims = gbcd.getComponentShape(); + + gbcdSizes[0] = static_cast(cDims[0]); + gbcdSizes[1] = static_cast(cDims[1]); + gbcdSizes[2] = static_cast(cDims[2]); + gbcdSizes[3] = static_cast(cDims[3]); + gbcdSizes[4] = static_cast(cDims[4]); + + gbcdDeltas[0] = (gbcdLimits[5] - gbcdLimits[0]) / static_cast(gbcdSizes[0]); + gbcdDeltas[1] = (gbcdLimits[6] - gbcdLimits[1]) / static_cast(gbcdSizes[1]); + gbcdDeltas[2] = (gbcdLimits[7] - gbcdLimits[2]) / static_cast(gbcdSizes[2]); + gbcdDeltas[3] = (gbcdLimits[8] - gbcdLimits[3]) / static_cast(gbcdSizes[3]); + gbcdDeltas[4] = (gbcdLimits[9] - gbcdLimits[4]) / static_cast(gbcdSizes[4]); + + int64 totalGbcdBins = gbcdSizes[0] * gbcdSizes[1] * gbcdSizes[2] * gbcdSizes[3] * gbcdSizes[4] * 2; + + // OOC optimization: cache only the phase-of-interest slice of the GBCD array. + // One phase = totalGbcdBins elements, bounded by bin resolution not array size. + const usize phaseOffset = static_cast(m_InputValues->PhaseOfInterest) * static_cast(totalGbcdBins); + auto gbcdPhaseCache = std::make_unique(static_cast(totalGbcdBins)); + gbcd.getDataStoreRef().copyIntoBuffer(phaseOffset, nonstd::span(gbcdPhaseCache.get(), static_cast(totalGbcdBins))); + + // Get our LaueOps pointer for the selected crystal structure + ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructuresCache[m_InputValues->PhaseOfInterest]]; + + int32 xPoints = m_InputValues->OutputImageDimension; + int32 yPoints = m_InputValues->OutputImageDimension; + + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Generating Intensity Plot for phase {} (OOC)", m_InputValues->PhaseOfInterest)}); + + // Parallel execution is safe — all data accessed through local cached buffers + ParallelData2DAlgorithm dataAlg; + dataAlg.setRange(0, xPoints, 0, yPoints); + + // Pass phaseOffset=0 since gbcdPhaseCache already points to the phase slice + dataAlg.execute(ComputeGBCDPoleFigureImpl(poleFigureCache.get(), {xPoints, yPoints}, orientOps, gbcdDeltas, gbcdLimits, gbcdSizes, gbcdPhaseCache.get(), 0, m_InputValues->MisorientationRotation)); + + // Write pole figure results back to the OOC store + poleFigure.getDataStoreRef().copyFromBuffer(0, nonstd::span(poleFigureCache.get(), poleFigureSize)); + + return {}; +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp new file mode 100644 index 0000000000..3945dd1d34 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp @@ -0,0 +1,43 @@ +#pragma once + +#include "OrientationAnalysis/OrientationAnalysis_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/VectorParameter.hpp" + +namespace nx::core +{ + +struct ComputeGBCDPoleFigureInputValues; + +/** + * @class ComputeGBCDPoleFigureScanline + * @brief OOC-safe variant that accesses the GBCD array through the DataStore + * without caching the entire array in RAM. Runs single-threaded since + * DataStore per-element access is not thread-safe for concurrent reads. + */ +class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureScanline +{ +public: + ComputeGBCDPoleFigureScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeGBCDPoleFigureInputValues* inputValues); + ~ComputeGBCDPoleFigureScanline() noexcept; + + ComputeGBCDPoleFigureScanline(const ComputeGBCDPoleFigureScanline&) = delete; + ComputeGBCDPoleFigureScanline(ComputeGBCDPoleFigureScanline&&) noexcept = delete; + ComputeGBCDPoleFigureScanline& operator=(const ComputeGBCDPoleFigureScanline&) = delete; + ComputeGBCDPoleFigureScanline& operator=(ComputeGBCDPoleFigureScanline&&) noexcept = delete; + + Result<> operator()(); + + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const ComputeGBCDPoleFigureInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp index 7c0d216acf..ff4e8d4988 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp @@ -1,131 +1,13 @@ #include "ComputeIPFColors.hpp" -#include "simplnx/Common/Array.hpp" -#include "simplnx/Common/RgbColor.hpp" -#include "simplnx/DataStructure/DataArray.hpp" -#include "simplnx/DataStructure/DataStore.hpp" -#include "simplnx/Utilities/ParallelDataAlgorithm.hpp" +#include "ComputeIPFColorsDirect.hpp" +#include "ComputeIPFColorsScanline.hpp" -#include -#include +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" using namespace nx::core; -namespace -{ - -/** - * @brief The ComputeIPFColorsImpl class implements a threaded algorithm that computes the IPF - * colors for each element in a geometry - */ -class ComputeIPFColorsImpl -{ -public: - ComputeIPFColorsImpl(ComputeIPFColors* filter, nx::core::FloatVec3 referenceDir, nx::core::Float32Array& eulers, nx::core::Int32Array& phases, nx::core::UInt32Array& crystalStructures, - int32_t numPhases, const nx::core::IDataArray* goodVoxels, nx::core::UInt8Array& colors) - : m_Filter(filter) - , m_ReferenceDir(referenceDir) - , m_CellEulerAngles(eulers.getDataStoreRef()) - , m_CellPhases(phases.getDataStoreRef()) - , m_CrystalStructures(crystalStructures.getDataStoreRef()) - , m_NumPhases(numPhases) - , m_GoodVoxels(goodVoxels) - , m_CellIPFColors(colors.getDataStoreRef()) - { - } - - virtual ~ComputeIPFColorsImpl() = default; - - template - void convert(size_t start, size_t end) const - { - using MaskArrayType = DataArray; - const MaskArrayType* maskArray = nullptr; - if(nullptr != m_GoodVoxels) - { - maskArray = dynamic_cast(m_GoodVoxels); - } - - std::vector ops = ebsdlib::LaueOps::GetAllOrientationOps(); - std::array refDir = {m_ReferenceDir[0], m_ReferenceDir[1], m_ReferenceDir[2]}; - std::array dEuler = {0.0, 0.0, 0.0}; - Rgba argb = 0x00000000; - int32_t phase = 0; - bool calcIPF = false; - size_t index = 0; - for(size_t i = start; i < end; i++) - { - if(m_Filter->shouldCancel()) - { - return; - } - phase = m_CellPhases[i]; - index = i * 3; - m_CellIPFColors.setValue(index, 0); - m_CellIPFColors.setValue(index + 1, 0); - m_CellIPFColors.setValue(index + 2, 0); - dEuler[0] = m_CellEulerAngles.getValue(index); - dEuler[1] = m_CellEulerAngles.getValue(index + 1); - dEuler[2] = m_CellEulerAngles.getValue(index + 2); - - // Make sure we are using a valid Euler Angles with valid crystal symmetry - calcIPF = true; - if(nullptr != maskArray) - { - calcIPF = (*maskArray)[i]; - } - // Sanity check the phase data to make sure we do not walk off the end of the array - if(phase >= m_NumPhases) - { - m_Filter->incrementPhaseWarningCount(); - } - - if(phase < m_NumPhases && calcIPF && m_CrystalStructures[phase] < ebsdlib::CrystalStructure::LaueGroupEnd) - { - argb = ops[m_CrystalStructures[phase]]->generateIPFColor(dEuler.data(), refDir.data(), false); - m_CellIPFColors.setValue(index, static_cast(nx::core::RgbColor::dRed(argb))); - m_CellIPFColors.setValue(index + 1, static_cast(nx::core::RgbColor::dGreen(argb))); - m_CellIPFColors.setValue(index + 2, static_cast(nx::core::RgbColor::dBlue(argb))); - } - } - } - - void run(size_t start, size_t end) const - { - if(m_GoodVoxels != nullptr) - { - if(m_GoodVoxels->getDataType() == DataType::boolean) - { - convert(start, end); - } - else if(m_GoodVoxels->getDataType() == DataType::uint8) - { - convert(start, end); - } - } - else - { - convert(start, end); - } - } - - void operator()(const Range& range) const - { - run(range.min(), range.max()); - } - -private: - ComputeIPFColors* m_Filter = nullptr; - nx::core::FloatVec3 m_ReferenceDir; - nx::core::Float32AbstractDataStore& m_CellEulerAngles; - nx::core::Int32AbstractDataStore& m_CellPhases; - nx::core::UInt32AbstractDataStore& m_CrystalStructures; - int32_t m_NumPhases = 0; - const nx::core::IDataArray* m_GoodVoxels = nullptr; - nx::core::UInt8AbstractDataStore& m_CellIPFColors; -}; -} // namespace - // ----------------------------------------------------------------------------- ComputeIPFColors::ComputeIPFColors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeIPFColorsInputValues* inputValues) : m_DataStructure(dataStructure) @@ -141,64 +23,9 @@ ComputeIPFColors::~ComputeIPFColors() noexcept = default; // ----------------------------------------------------------------------------- Result<> ComputeIPFColors::operator()() { + auto* eulersArray = m_DataStructure.getDataAs(m_InputValues->cellEulerAnglesArrayPath); + auto* phasesArray = m_DataStructure.getDataAs(m_InputValues->cellPhasesArrayPath); + auto* ipfColorsArray = m_DataStructure.getDataAs(m_InputValues->cellIpfColorsArrayPath); - std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - - nx::core::Float32Array& eulers = m_DataStructure.getDataRefAs(m_InputValues->cellEulerAnglesArrayPath); - nx::core::Int32Array& phases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); - - nx::core::UInt32Array& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); - - nx::core::UInt8Array& ipfColors = m_DataStructure.getDataRefAs(m_InputValues->cellIpfColorsArrayPath); - - m_PhaseWarningCount = 0; - size_t totalPoints = eulers.getNumberOfTuples(); - - int32_t numPhases = static_cast(crystalStructures.getNumberOfTuples()); - - // Make sure we are dealing with a unit 1 vector. - nx::core::FloatVec3 normRefDir = m_InputValues->referenceDirection; // Make a copy of the reference Direction - normRefDir = normRefDir.normalize(); - - typename IParallelAlgorithm::AlgorithmArrays algArrays; - algArrays.push_back(&eulers); - algArrays.push_back(&phases); - algArrays.push_back(&crystalStructures); - algArrays.push_back(&ipfColors); - - nx::core::IDataArray* goodVoxelsArray = nullptr; - if(m_InputValues->useGoodVoxels) - { - goodVoxelsArray = m_DataStructure.getDataAs(m_InputValues->goodVoxelsArrayPath); - algArrays.push_back(goodVoxelsArray); - } - - // Allow data-based parallelization - ParallelDataAlgorithm dataAlg; - dataAlg.setRange(0, totalPoints); - dataAlg.requireArraysInMemory(algArrays); - - dataAlg.execute(ComputeIPFColorsImpl(this, normRefDir, eulers, phases, crystalStructures, numPhases, goodVoxelsArray, ipfColors)); - - if(m_PhaseWarningCount > 0) - { - std::string message = fmt::format("The Ensemble Phase information only references {} phase(s) but {} cell(s) had a phase value greater than {}. \ -This indicates a problem with the input cell phase data. DREAM3D-NX will give INCORRECT RESULTS.", - (numPhases - 1), m_PhaseWarningCount, (numPhases - 1)); - - return nx::core::MakeErrorResult(-48000, message); - } - - return {}; -} - -// ----------------------------------------------------------------------------- -void ComputeIPFColors::incrementPhaseWarningCount() -{ - ++m_PhaseWarningCount; -} - -bool ComputeIPFColors::shouldCancel() const -{ - return m_ShouldCancel; + return DispatchAlgorithm({eulersArray, phasesArray, ipfColorsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp index 7087c2b300..f72cf32f5b 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp @@ -2,10 +2,8 @@ #include "OrientationAnalysis/OrientationAnalysis_export.hpp" -#include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" -#include "simplnx/DataStructure/IDataArray.hpp" #include "simplnx/Filter/IFilter.hpp" #include @@ -13,13 +11,10 @@ namespace nx::core { -/** - * @brief The ComputeIPFColorsInputValues struct - */ struct ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsInputValues { std::vector referenceDirection; - bool useGoodVoxels; + bool useGoodVoxels = false; DataPath goodVoxelsArrayPath; DataPath cellPhasesArrayPath; DataPath cellEulerAnglesArrayPath; @@ -27,36 +22,24 @@ struct ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsInputValues DataPath cellIpfColorsArrayPath; }; -/** - * @brief - */ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColors { public: ComputeIPFColors(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ComputeIPFColorsInputValues* inputValues); ~ComputeIPFColors() noexcept; - ComputeIPFColors(const ComputeIPFColors&) = delete; // Copy Constructor Not Implemented - ComputeIPFColors(ComputeIPFColors&&) = delete; // Move Constructor Not Implemented - ComputeIPFColors& operator=(const ComputeIPFColors&) = delete; // Copy Assignment Not Implemented - ComputeIPFColors& operator=(ComputeIPFColors&&) = delete; // Move Assignment Not Implemented + ComputeIPFColors(const ComputeIPFColors&) = delete; + ComputeIPFColors(ComputeIPFColors&&) = delete; + ComputeIPFColors& operator=(const ComputeIPFColors&) = delete; + ComputeIPFColors& operator=(ComputeIPFColors&&) = delete; Result<> operator()(); - /** - * @brief incrementPhaseWarningCount - */ - void incrementPhaseWarningCount(); - - bool shouldCancel() const; - private: DataStructure& m_DataStructure; const IFilter::MessageHandler& m_MessageHandler; const std::atomic_bool& m_ShouldCancel; const ComputeIPFColorsInputValues* m_InputValues = nullptr; - - int32_t m_PhaseWarningCount = 0; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp new file mode 100644 index 0000000000..1e9bac69fc --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp @@ -0,0 +1,206 @@ +#include "ComputeIPFColorsDirect.hpp" + +#include "ComputeIPFColors.hpp" + +#include "simplnx/Common/Array.hpp" +#include "simplnx/Common/RgbColor.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataStore.hpp" +#include "simplnx/Utilities/ParallelDataAlgorithm.hpp" + +#include +#include + +using namespace nx::core; + +namespace +{ + +/** + * @brief The ComputeIPFColorsImpl class implements a threaded algorithm that computes the IPF + * colors for each element in a geometry + */ +class ComputeIPFColorsImpl +{ +public: + ComputeIPFColorsImpl(ComputeIPFColorsDirect* filter, nx::core::FloatVec3 referenceDir, nx::core::Float32Array& eulers, nx::core::Int32Array& phases, nx::core::UInt32Array& crystalStructures, + int32_t numPhases, const nx::core::IDataArray* goodVoxels, nx::core::UInt8Array& colors) + : m_Filter(filter) + , m_ReferenceDir(referenceDir) + , m_CellEulerAngles(eulers.getDataStoreRef()) + , m_CellPhases(phases.getDataStoreRef()) + , m_CrystalStructures(crystalStructures.getDataStoreRef()) + , m_NumPhases(numPhases) + , m_GoodVoxels(goodVoxels) + , m_CellIPFColors(colors.getDataStoreRef()) + { + } + + virtual ~ComputeIPFColorsImpl() = default; + + template + void convert(size_t start, size_t end) const + { + using MaskArrayType = DataArray; + const MaskArrayType* maskArray = nullptr; + if(nullptr != m_GoodVoxels) + { + maskArray = dynamic_cast(m_GoodVoxels); + } + + std::vector ops = ebsdlib::LaueOps::GetAllOrientationOps(); + std::array refDir = {m_ReferenceDir[0], m_ReferenceDir[1], m_ReferenceDir[2]}; + std::array dEuler = {0.0, 0.0, 0.0}; + Rgba argb = 0x00000000; + int32_t phase = 0; + bool calcIPF = false; + size_t index = 0; + for(size_t i = start; i < end; i++) + { + if(m_Filter->shouldCancel()) + { + return; + } + phase = m_CellPhases[i]; + index = i * 3; + m_CellIPFColors.setValue(index, 0); + m_CellIPFColors.setValue(index + 1, 0); + m_CellIPFColors.setValue(index + 2, 0); + dEuler[0] = m_CellEulerAngles.getValue(index); + dEuler[1] = m_CellEulerAngles.getValue(index + 1); + dEuler[2] = m_CellEulerAngles.getValue(index + 2); + + // Make sure we are using a valid Euler Angles with valid crystal symmetry + calcIPF = true; + if(nullptr != maskArray) + { + calcIPF = (*maskArray)[i]; + } + // Sanity check the phase data to make sure we do not walk off the end of the array + if(phase >= m_NumPhases) + { + m_Filter->incrementPhaseWarningCount(); + } + + if(phase < m_NumPhases && calcIPF && m_CrystalStructures[phase] < ebsdlib::CrystalStructure::LaueGroupEnd) + { + argb = ops[m_CrystalStructures[phase]]->generateIPFColor(dEuler.data(), refDir.data(), false); + m_CellIPFColors.setValue(index, static_cast(nx::core::RgbColor::dRed(argb))); + m_CellIPFColors.setValue(index + 1, static_cast(nx::core::RgbColor::dGreen(argb))); + m_CellIPFColors.setValue(index + 2, static_cast(nx::core::RgbColor::dBlue(argb))); + } + } + } + + void run(size_t start, size_t end) const + { + if(m_GoodVoxels != nullptr) + { + if(m_GoodVoxels->getDataType() == DataType::boolean) + { + convert(start, end); + } + else if(m_GoodVoxels->getDataType() == DataType::uint8) + { + convert(start, end); + } + } + else + { + convert(start, end); + } + } + + void operator()(const Range& range) const + { + run(range.min(), range.max()); + } + +private: + ComputeIPFColorsDirect* m_Filter = nullptr; + nx::core::FloatVec3 m_ReferenceDir; + nx::core::Float32AbstractDataStore& m_CellEulerAngles; + nx::core::Int32AbstractDataStore& m_CellPhases; + nx::core::UInt32AbstractDataStore& m_CrystalStructures; + int32_t m_NumPhases = 0; + const nx::core::IDataArray* m_GoodVoxels = nullptr; + nx::core::UInt8AbstractDataStore& m_CellIPFColors; +}; +} // namespace + +// ----------------------------------------------------------------------------- +ComputeIPFColorsDirect::ComputeIPFColorsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeIPFColorsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_MessageHandler(mesgHandler) +, m_ShouldCancel(shouldCancel) +, m_InputValues(inputValues) +{ +} + +// ----------------------------------------------------------------------------- +ComputeIPFColorsDirect::~ComputeIPFColorsDirect() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> ComputeIPFColorsDirect::operator()() +{ + std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + + nx::core::Float32Array& eulers = m_DataStructure.getDataRefAs(m_InputValues->cellEulerAnglesArrayPath); + nx::core::Int32Array& phases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); + + nx::core::UInt32Array& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); + + nx::core::UInt8Array& ipfColors = m_DataStructure.getDataRefAs(m_InputValues->cellIpfColorsArrayPath); + + m_PhaseWarningCount = 0; + size_t totalPoints = eulers.getNumberOfTuples(); + + int32_t numPhases = static_cast(crystalStructures.getNumberOfTuples()); + + // Make sure we are dealing with a unit 1 vector. + nx::core::FloatVec3 normRefDir = m_InputValues->referenceDirection; // Make a copy of the reference Direction + normRefDir = normRefDir.normalize(); + + typename IParallelAlgorithm::AlgorithmArrays algArrays; + algArrays.push_back(&eulers); + algArrays.push_back(&phases); + algArrays.push_back(&crystalStructures); + algArrays.push_back(&ipfColors); + + nx::core::IDataArray* goodVoxelsArray = nullptr; + if(m_InputValues->useGoodVoxels) + { + goodVoxelsArray = m_DataStructure.getDataAs(m_InputValues->goodVoxelsArrayPath); + algArrays.push_back(goodVoxelsArray); + } + + // Allow data-based parallelization + ParallelDataAlgorithm dataAlg; + dataAlg.setRange(0, totalPoints); + dataAlg.requireArraysInMemory(algArrays); + + dataAlg.execute(ComputeIPFColorsImpl(this, normRefDir, eulers, phases, crystalStructures, numPhases, goodVoxelsArray, ipfColors)); + + if(m_PhaseWarningCount > 0) + { + std::string message = fmt::format("The Ensemble Phase information only references {} phase(s) but {} cell(s) had a phase value greater than {}. \ +This indicates a problem with the input cell phase data. DREAM3D-NX will give INCORRECT RESULTS.", + (numPhases - 1), m_PhaseWarningCount, (numPhases - 1)); + + return nx::core::MakeErrorResult(-48000, message); + } + + return {}; +} + +// ----------------------------------------------------------------------------- +void ComputeIPFColorsDirect::incrementPhaseWarningCount() +{ + ++m_PhaseWarningCount; +} + +bool ComputeIPFColorsDirect::shouldCancel() const +{ + return m_ShouldCancel; +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp new file mode 100644 index 0000000000..b17a926628 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp @@ -0,0 +1,38 @@ +#pragma once + +#include "OrientationAnalysis/OrientationAnalysis_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct ComputeIPFColorsInputValues; + +class ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsDirect +{ +public: + ComputeIPFColorsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, const ComputeIPFColorsInputValues* inputValues); + ~ComputeIPFColorsDirect() noexcept; + + ComputeIPFColorsDirect(const ComputeIPFColorsDirect&) = delete; + ComputeIPFColorsDirect(ComputeIPFColorsDirect&&) = delete; + ComputeIPFColorsDirect& operator=(const ComputeIPFColorsDirect&) = delete; + ComputeIPFColorsDirect& operator=(ComputeIPFColorsDirect&&) = delete; + + Result<> operator()(); + + void incrementPhaseWarningCount(); + bool shouldCancel() const; + +private: + DataStructure& m_DataStructure; + const IFilter::MessageHandler& m_MessageHandler; + const std::atomic_bool& m_ShouldCancel; + const ComputeIPFColorsInputValues* m_InputValues = nullptr; + int32_t m_PhaseWarningCount = 0; +}; + +} // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp new file mode 100644 index 0000000000..dfde285e6a --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp @@ -0,0 +1,166 @@ +#include "ComputeIPFColorsScanline.hpp" + +#include "ComputeIPFColors.hpp" + +#include "simplnx/Common/Array.hpp" +#include "simplnx/Common/RgbColor.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataStore.hpp" + +#include + +#include + +#include + +using namespace nx::core; + +namespace +{ +constexpr usize k_ChunkTuples = 65536; +} // namespace + +// ----------------------------------------------------------------------------- +ComputeIPFColorsScanline::ComputeIPFColorsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeIPFColorsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_MessageHandler(mesgHandler) +, m_ShouldCancel(shouldCancel) +, m_InputValues(inputValues) +{ +} + +// ----------------------------------------------------------------------------- +ComputeIPFColorsScanline::~ComputeIPFColorsScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> ComputeIPFColorsScanline::operator()() +{ + std::vector ops = ebsdlib::LaueOps::GetAllOrientationOps(); + + auto& eulers = m_DataStructure.getDataRefAs(m_InputValues->cellEulerAnglesArrayPath); + auto& phases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); + auto& crystalStructuresArray = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); + auto& ipfColors = m_DataStructure.getDataRefAs(m_InputValues->cellIpfColorsArrayPath); + + const usize totalPoints = eulers.getNumberOfTuples(); + const int32 numPhases = static_cast(crystalStructuresArray.getNumberOfTuples()); + + // Cache crystal structures locally (ensemble-level, tiny) + std::vector crystalStructures(numPhases); + crystalStructuresArray.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructures.data(), static_cast(numPhases))); + + // Normalize reference direction + FloatVec3 normRefDir = m_InputValues->referenceDirection; + normRefDir = normRefDir.normalize(); + std::array refDir = {normRefDir[0], normRefDir[1], normRefDir[2]}; + + // Optional mask array + const IDataArray* goodVoxelsArray = nullptr; + if(m_InputValues->useGoodVoxels) + { + goodVoxelsArray = m_DataStructure.getDataAs(m_InputValues->goodVoxelsArrayPath); + } + + const auto& eulersStore = eulers.getDataStoreRef(); + const auto& phasesStore = phases.getDataStoreRef(); + auto& ipfColorsStore = ipfColors.getDataStoreRef(); + + // Allocate chunk buffers + auto eulerBuf = std::make_unique(k_ChunkTuples * 3); + auto phasesBuf = std::make_unique(k_ChunkTuples); + auto colorBuf = std::make_unique(k_ChunkTuples * 3); + + // Mask buffer — only allocated if needed + std::unique_ptr boolMaskBuf; + std::unique_ptr uint8MaskBuf; + const bool hasBoolMask = goodVoxelsArray != nullptr && goodVoxelsArray->getDataType() == DataType::boolean; + const bool hasUint8Mask = goodVoxelsArray != nullptr && goodVoxelsArray->getDataType() == DataType::uint8; + if(hasBoolMask) + { + boolMaskBuf = std::make_unique(k_ChunkTuples); + } + else if(hasUint8Mask) + { + uint8MaskBuf = std::make_unique(k_ChunkTuples); + } + + int32 phaseWarningCount = 0; + std::array dEuler = {0.0, 0.0, 0.0}; + + for(usize offset = 0; offset < totalPoints; offset += k_ChunkTuples) + { + if(m_ShouldCancel) + { + return {}; + } + + const usize count = std::min(k_ChunkTuples, totalPoints - offset); + + // Bulk read cell-level data + eulersStore.copyIntoBuffer(offset * 3, nonstd::span(eulerBuf.get(), count * 3)); + phasesStore.copyIntoBuffer(offset, nonstd::span(phasesBuf.get(), count)); + + // Read mask chunk if applicable + if(hasBoolMask) + { + dynamic_cast(goodVoxelsArray)->getDataStoreRef().copyIntoBuffer(offset, nonstd::span(boolMaskBuf.get(), count)); + } + else if(hasUint8Mask) + { + dynamic_cast(goodVoxelsArray)->getDataStoreRef().copyIntoBuffer(offset, nonstd::span(uint8MaskBuf.get(), count)); + } + + // Process chunk + for(usize i = 0; i < count; i++) + { + const usize ci = i * 3; + // Default to black + colorBuf[ci] = 0; + colorBuf[ci + 1] = 0; + colorBuf[ci + 2] = 0; + + const int32 phase = phasesBuf[i]; + dEuler[0] = eulerBuf[ci]; + dEuler[1] = eulerBuf[ci + 1]; + dEuler[2] = eulerBuf[ci + 2]; + + bool calcIPF = true; + if(hasBoolMask) + { + calcIPF = boolMaskBuf[i]; + } + else if(hasUint8Mask) + { + calcIPF = uint8MaskBuf[i] != 0; + } + + if(phase >= numPhases) + { + phaseWarningCount++; + } + + if(phase < numPhases && calcIPF && crystalStructures[phase] < ebsdlib::CrystalStructure::LaueGroupEnd) + { + Rgba argb = ops[crystalStructures[phase]]->generateIPFColor(dEuler.data(), refDir.data(), false); + colorBuf[ci] = static_cast(RgbColor::dRed(argb)); + colorBuf[ci + 1] = static_cast(RgbColor::dGreen(argb)); + colorBuf[ci + 2] = static_cast(RgbColor::dBlue(argb)); + } + } + + // Bulk write colors + ipfColorsStore.copyFromBuffer(offset * 3, nonstd::span(colorBuf.get(), count * 3)); + } + + if(phaseWarningCount > 0) + { + std::string message = fmt::format("The Ensemble Phase information only references {} phase(s) but {} cell(s) had a phase value greater than {}. " + "This indicates a problem with the input cell phase data. DREAM3D-NX will give INCORRECT RESULTS.", + (numPhases - 1), phaseWarningCount, (numPhases - 1)); + + return nx::core::MakeErrorResult(-48000, message); + } + + return {}; +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp new file mode 100644 index 0000000000..ba633a8af2 --- /dev/null +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp @@ -0,0 +1,34 @@ +#pragma once + +#include "OrientationAnalysis/OrientationAnalysis_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct ComputeIPFColorsInputValues; + +class ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsScanline +{ +public: + ComputeIPFColorsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, const ComputeIPFColorsInputValues* inputValues); + ~ComputeIPFColorsScanline() noexcept; + + ComputeIPFColorsScanline(const ComputeIPFColorsScanline&) = delete; + ComputeIPFColorsScanline(ComputeIPFColorsScanline&&) = delete; + ComputeIPFColorsScanline& operator=(const ComputeIPFColorsScanline&) = delete; + ComputeIPFColorsScanline& operator=(ComputeIPFColorsScanline&&) = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const IFilter::MessageHandler& m_MessageHandler; + const std::atomic_bool& m_ShouldCancel; + const ComputeIPFColorsInputValues* m_InputValues = nullptr; +}; + +} // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp index ab80ebb2de..13b221c34e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp @@ -2,199 +2,174 @@ #include "simplnx/Common/Constants.hpp" #include "simplnx/DataStructure/DataArray.hpp" -#include "simplnx/DataStructure/DataGroup.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" -#include "simplnx/Utilities/MessageHelper.hpp" -#include "simplnx/Utilities/ParallelData3DAlgorithm.hpp" #include -#include +#include using namespace nx::core; -namespace +// ----------------------------------------------------------------------------- +ComputeKernelAvgMisorientations::ComputeKernelAvgMisorientations(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeKernelAvgMisorientationsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) { -class FindKernelAvgMisorientationsImpl +} + +// ----------------------------------------------------------------------------- +ComputeKernelAvgMisorientations::~ComputeKernelAvgMisorientations() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> ComputeKernelAvgMisorientations::operator()() { -public: - FindKernelAvgMisorientationsImpl(ProgressMessageHelper& progressMessenger, DataStructure& dataStructure, const ComputeKernelAvgMisorientationsInputValues* inputValues, - const std::atomic_bool& shouldCancel) - : m_ProgressMessageHelper(progressMessenger) - , m_DataStructure(dataStructure) - , m_InputValues(inputValues) - , m_ShouldCancel(shouldCancel) - { - } + auto* gridGeom = m_DataStructure.getDataAs(m_InputValues->InputImageGeometry); + SizeVec3 udims = gridGeom->getDimensions(); - void convert(size_t zStart, size_t zEnd, size_t yStart, size_t yEnd, size_t xStart, size_t xEnd) const + const auto xPoints = static_cast(udims[0]); + const auto yPoints = static_cast(udims[1]); + const auto zPoints = static_cast(udims[2]); + const usize sliceSize = static_cast(xPoints * yPoints); + const auto kernelSize = m_InputValues->KernelSize; + const int32 kZ = kernelSize[2]; + const int32 kY = kernelSize[1]; + const int32 kX = kernelSize[0]; + + // Get DataStore references for bulk I/O + const auto& cellPhasesStore = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath).getDataStoreRef(); + const auto& featureIdsStore = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath).getDataStoreRef(); + const auto& quatsStore = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath).getDataStoreRef(); + auto& outputStore = m_DataStructure.getDataRefAs(m_InputValues->KernelAverageMisorientationsArrayName).getDataStoreRef(); + + // Cache ensemble-level crystalStructures locally (tiny array, avoids per-element OOC overhead) + const auto& crystalStructuresStore = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath).getDataStoreRef(); + const usize numCrystalStructures = crystalStructuresStore.getNumberOfTuples(); + std::vector crystalStructuresLocal(numCrystalStructures); + crystalStructuresStore.copyIntoBuffer(0, nonstd::span(crystalStructuresLocal.data(), numCrystalStructures)); + + std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + + // Slab-based processing: for each Z-plane, read a slab of input data + // spanning [plane - kZ, plane + kZ] in Z. This covers all neighbor + // lookups for voxels in this plane. + // + // Slab buffers hold (slabZCount * Y * X) elements for 1-component arrays + // and (slabZCount * Y * X * 4) for quaternions. + + for(int64 plane = 0; plane < zPoints; plane++) { - // Input Arrays / Parameter Data - const auto& cellPhasesArray = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); - const auto& cellPhases = cellPhasesArray.getDataStoreRef(); - const auto& featureIdsArray = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); - const auto& featureIds = featureIdsArray.getDataStoreRef(); - const auto& quatsArray = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); - const auto& quats = quatsArray.getDataStoreRef(); - const auto& crystalStructuresArray = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); - const auto& crystalStructures = crystalStructuresArray.getDataStoreRef(); - const auto kernelSize = m_InputValues->KernelSize; - - // Output Arrays - auto& kernelAvgMisorientationsArray = m_DataStructure.getDataRefAs(m_InputValues->KernelAverageMisorientationsArrayName); - auto& kernelAvgMisorientations = kernelAvgMisorientationsArray.getDataStoreRef(); - - std::vector m_OrientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - - auto* gridGeom = m_DataStructure.getDataAs(m_InputValues->InputImageGeometry); - SizeVec3 udims = gridGeom->getDimensions(); - - ebsdlib::QuatD q1; - ebsdlib::QuatD q2; - - // messenger values - usize counter = 0; - usize increment = (zEnd - zStart) / 100; - - ProgressMessenger progressMessenger = m_ProgressMessageHelper.createProgressMessenger(); - - auto xPoints = static_cast(udims[0]); - auto yPoints = static_cast(udims[1]); - auto zPoints = static_cast(udims[2]); - for(size_t plane = zStart; plane < zEnd; plane++) + if(m_ShouldCancel) { - if(m_ShouldCancel) - { - break; - } + break; + } + // Compute slab Z range (clamped to volume bounds) + const int64 slabZMin = std::max(static_cast(0), plane - kZ); + const int64 slabZMax = std::min(zPoints - 1, plane + kZ); + const usize slabZCount = static_cast(slabZMax - slabZMin + 1); + const usize slabTuples = slabZCount * sliceSize; - if(counter > increment) - { - progressMessenger.sendProgressMessage(counter); - counter = 0; - } + // Read slab data via copyIntoBuffer (OOC-safe bulk I/O) + const usize slabStartTuple = static_cast(slabZMin) * sliceSize; + + std::vector slabFeatureIds(slabTuples); + featureIdsStore.copyIntoBuffer(slabStartTuple, nonstd::span(slabFeatureIds.data(), slabTuples)); + + std::vector slabCellPhases(slabTuples); + cellPhasesStore.copyIntoBuffer(slabStartTuple, nonstd::span(slabCellPhases.data(), slabTuples)); - for(size_t row = yStart; row < yEnd; row++) + std::vector slabQuats(slabTuples * 4); + quatsStore.copyIntoBuffer(slabStartTuple * 4, nonstd::span(slabQuats.data(), slabTuples * 4)); + + // Output buffer for this plane + std::vector planeOutput(sliceSize, 0.0f); + + // Offset of current plane within the slab + const usize planeOffsetInSlab = static_cast(plane - slabZMin) * sliceSize; + + for(int64 row = 0; row < yPoints; row++) + { + for(int64 col = 0; col < xPoints; col++) { - for(size_t col = xStart; col < xEnd; col++) + const usize pointInSlab = planeOffsetInSlab + static_cast(row * xPoints + col); + const usize pointInPlane = static_cast(row * xPoints + col); + + const int32 featureId = slabFeatureIds[pointInSlab]; + const int32 cellPhase = slabCellPhases[pointInSlab]; + + if(featureId <= 0 || cellPhase <= 0) { - size_t point = (plane * xPoints * yPoints) + (row * xPoints) + col; - if(featureIds[point] > 0 && cellPhases[point] > 0) - { - float totalMisorientation = 0.0f; - int32 numVoxel = 0; + planeOutput[pointInPlane] = 0.0f; + continue; + } + + // Extract center quaternion + ebsdlib::QuatD q1; + const usize q1Idx = pointInSlab * 4; + q1[0] = slabQuats[q1Idx]; + q1[1] = slabQuats[q1Idx + 1]; + q1[2] = slabQuats[q1Idx + 2]; + q1[3] = slabQuats[q1Idx + 3]; + + const uint32 laueClass = crystalStructuresLocal[static_cast(cellPhase)]; - size_t quatIndex = point * 4; - q1[0] = quats[quatIndex]; - q1[1] = quats[quatIndex + 1]; - q1[2] = quats[quatIndex + 2]; - q1[3] = quats[quatIndex + 3]; + float32 totalMisorientation = 0.0f; + int32 numVoxel = 0; + + for(int32 j = -kZ; j <= kZ; j++) + { + const int64 nz = plane + j; + if(nz < 0 || nz >= zPoints) + { + continue; + } + const usize nzInSlab = static_cast(nz - slabZMin) * sliceSize; - for(int32_t j = -kernelSize[2]; j < kernelSize[2] + 1; j++) + for(int32 k = -kY; k <= kY; k++) + { + const int64 ny = row + k; + if(ny < 0 || ny >= yPoints) { + continue; + } - if(plane + j < 0 || plane + j > zPoints - 1) + for(int32 l = -kX; l <= kX; l++) + { + const int64 nx = col + l; + if(nx < 0 || nx >= xPoints) { continue; } - const int64_t jStride = j * xPoints * yPoints; - for(int32_t k = -kernelSize[1]; k < kernelSize[1] + 1; k++) + + const usize neighborInSlab = nzInSlab + static_cast(ny * xPoints + nx); + + if(slabFeatureIds[neighborInSlab] == featureId) { - if(row + k < 0 || row + k > yPoints - 1) - { - continue; - } - const int64_t kStride = k * xPoints; - for(int32_t l = -kernelSize[0]; l < kernelSize[0] + 1; l++) - { - if(col + l < 0 || col + l > xPoints - 1) - { - continue; - } - const int64_t neighbor = static_cast(point) + jStride + kStride + l; - if(neighbor >= 0 && featureIds[point] == featureIds[static_cast(neighbor)]) - { - quatIndex = neighbor * 4; - q2[0] = quats[quatIndex]; - q2[1] = quats[quatIndex + 1]; - q2[2] = quats[quatIndex + 2]; - q2[3] = quats[quatIndex + 3]; - uint32_t laueClass = crystalStructures[cellPhases[point]]; - ebsdlib::AxisAngleDType axisAngle = m_OrientationOps[laueClass]->calculateMisorientation(q1, q2); - totalMisorientation = totalMisorientation + (axisAngle[3] * nx::core::Constants::k_180OverPiF); - numVoxel++; - } - } + const usize q2Idx = neighborInSlab * 4; + ebsdlib::QuatD q2; + q2[0] = slabQuats[q2Idx]; + q2[1] = slabQuats[q2Idx + 1]; + q2[2] = slabQuats[q2Idx + 2]; + q2[3] = slabQuats[q2Idx + 3]; + + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass]->calculateMisorientation(q1, q2); + totalMisorientation += (axisAngle[3] * nx::core::Constants::k_180OverPiF); + numVoxel++; } } - kernelAvgMisorientations[point] = totalMisorientation / static_cast(numVoxel); - if(numVoxel == 0) - { - kernelAvgMisorientations[point] = 0.0f; - } } - if(featureIds[point] == 0 || cellPhases[point] == 0) - { - kernelAvgMisorientations[point] = 0.0f; - } - - counter++; } + + planeOutput[pointInPlane] = (numVoxel > 0) ? (totalMisorientation / static_cast(numVoxel)) : 0.0f; } } - progressMessenger.sendProgressMessage(counter); - } - void operator()(const Range3D& range) const - { - convert(range[4], range[5], range[2], range[3], range[0], range[1]); + // Write this plane's output via bulk I/O + const usize planeStartTuple = static_cast(plane) * sliceSize; + outputStore.copyFromBuffer(planeStartTuple, nonstd::span(planeOutput.data(), sliceSize)); } -private: - ProgressMessageHelper& m_ProgressMessageHelper; - DataStructure& m_DataStructure; - const ComputeKernelAvgMisorientationsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; -}; - -} // namespace - -// ----------------------------------------------------------------------------- -ComputeKernelAvgMisorientations::ComputeKernelAvgMisorientations(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - ComputeKernelAvgMisorientationsInputValues* inputValues) -: m_DataStructure(dataStructure) -, m_InputValues(inputValues) -, m_ShouldCancel(shouldCancel) -, m_MessageHandler(mesgHandler) -{ -} - -// ----------------------------------------------------------------------------- -ComputeKernelAvgMisorientations::~ComputeKernelAvgMisorientations() noexcept = default; - -// ----------------------------------------------------------------------------- -Result<> ComputeKernelAvgMisorientations::operator()() -{ - auto* gridGeom = m_DataStructure.getDataAs(m_InputValues->InputImageGeometry); - SizeVec3 udims = gridGeom->getDimensions(); - - MessageHelper messageHelper(m_MessageHandler); - ProgressMessageHelper progressMessageHelper = messageHelper.createProgressMessageHelper(); - - progressMessageHelper.setMaxProgresss(udims[2] * udims[1] * udims[0]); - progressMessageHelper.setProgressMessageTemplate("Finding Kernel Average Misorientations || {:.2f}%"); - - typename IParallelAlgorithm::AlgorithmArrays algArrays; - algArrays.push_back(m_DataStructure.getDataAs(m_InputValues->CellPhasesArrayPath)); - algArrays.push_back(m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)); - algArrays.push_back(m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)); - algArrays.push_back(m_DataStructure.getDataAs(m_InputValues->KernelAverageMisorientationsArrayName)); - algArrays.push_back(m_DataStructure.getDataAs(m_InputValues->QuatsArrayPath)); - - ParallelData3DAlgorithm parallelAlgorithm; - parallelAlgorithm.setRange(Range3D(0, udims[0], 0, udims[1], 0, udims[2])); - parallelAlgorithm.requireArraysInMemory(algArrays); - parallelAlgorithm.execute(FindKernelAvgMisorientationsImpl(progressMessageHelper, m_DataStructure, m_InputValues, m_ShouldCancel)); - return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp index 73f9647ac4..492409f5fc 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp @@ -12,6 +12,7 @@ #include +#include #include using namespace nx::core; @@ -43,12 +44,10 @@ bool IsTwinBoundary(const Eigen::Quaternion& quat1, const Eigen::Quaternion jQuat = orientationOps[laueClass]->getQuatSymOp(j); sym_q = Eigen::Quaterniond(jQuat.w(), jQuat.x(), jQuat.y(), jQuat.z()); - // calculate crystal direction parallel to normal s1_misq = misq * sym_q; for(int32 k = 0; k < nsym; k++) { - // calculate the symmetric misorienation ebsdlib::Quaternion kQuat = orientationOps[laueClass]->getQuatSymOp(k); sym_q = Eigen::Quaterniond(kQuat.w(), kQuat.x(), kQuat.y(), kQuat.z()); sym_q = sym_q.conjugate(); @@ -103,12 +102,10 @@ std::optional FindTwinBoundaryIncoherence(const Eigen::Vector3d& xstl_norm, c ebsdlib::Quaternion jQuat = orientationOps[laueClass]->getQuatSymOp(j); j_sym_q = Eigen::Quaterniond(jQuat.w(), jQuat.x(), jQuat.y(), jQuat.z()); - // calculate crystal direction parallel to normal s1_misq = misq * j_sym_q; for(int32 k = 0; k < nsym; k++) { - // calculate the symmetric misorienation ebsdlib::Quaternion kQuat = orientationOps[laueClass]->getQuatSymOp(k); sym_q = Eigen::Quaterniond(kQuat.w(), kQuat.x(), kQuat.y(), kQuat.z()); sym_q = sym_q.conjugate(); @@ -128,7 +125,7 @@ std::optional FindTwinBoundaryIncoherence(const Eigen::Vector3d& xstl_norm, c if(axisdiff111 < axisTolerance && angdiff60 < angTolerance) { const Eigen::Vector3d axVec{xVal, yVal, zVal}; - const Eigen::Vector3d s_xstl_norm = j_sym_q.conjugate()._transformVector(xstl_norm); // conjugate for active rotate + const Eigen::Vector3d s_xstl_norm = j_sym_q.conjugate()._transformVector(xstl_norm); T incoherence = 180.0 * std::acos(GeometryMath::CosThetaBetweenVectors(axVec, s_xstl_norm)) / nx::core::Constants::k_PiD; if(incoherence > 90.0) @@ -153,18 +150,17 @@ std::optional FindTwinBoundaryIncoherence(const Eigen::Vector3d& xstl_norm, c } /** - * @brief The CalculateTwinBoundaryImpl class implements a threaded algorithm that determines whether a boundary is twin related and calculates - * the respective incoherence. The calculations are performed on a surface mesh. + * @brief Parallel worker for twin boundaries with incoherence. + * All arrays are locally cached vectors — zero OOC virtual dispatch in the hot loop. */ class CalculateTwinBoundaryWithIncoherenceImpl { using Matrix3x3 = Eigen::Matrix; public: - CalculateTwinBoundaryWithIncoherenceImpl(float32 angtol, float32 axistol, const Int32AbstractDataStore& faceLabels, const Float64AbstractDataStore& faceNormals, - const Float32AbstractDataStore& avgQuats, const Int32AbstractDataStore& featurePhases, const UInt32AbstractDataStore& crystalStructures, - std::unique_ptr& twinBoundaries, Float32AbstractDataStore& twinBoundaryIncoherence, const std::atomic_bool& shouldCancel, - std::atomic_bool& hasNaN) + CalculateTwinBoundaryWithIncoherenceImpl(float32 angtol, float32 axistol, const std::vector& faceLabels, const std::vector& faceNormals, const std::vector& avgQuats, + const std::vector& featurePhases, const std::vector& crystalStructures, std::vector& twinBoundariesOut, + std::vector& twinBoundaryIncoherenceOut, const std::atomic_bool& shouldCancel, std::atomic_bool& hasNaN) : m_AxisTol(axistol) , m_AngTol(angtol) , m_FaceLabels(faceLabels) @@ -172,8 +168,8 @@ class CalculateTwinBoundaryWithIncoherenceImpl , m_AvgQuats(avgQuats) , m_FeaturePhases(featurePhases) , m_CrystalStructures(crystalStructures) - , m_TwinBoundaries(twinBoundaries) - , m_TwinBoundaryIncoherence(twinBoundaryIncoherence) + , m_TwinBoundariesOut(twinBoundariesOut) + , m_TwinBoundaryIncoherenceOut(twinBoundaryIncoherenceOut) , m_ShouldCancel(shouldCancel) , m_HasNaN(hasNaN) , m_OrientationOps(ebsdlib::LaueOps::GetAllOrientationOps()) @@ -190,21 +186,20 @@ class CalculateTwinBoundaryWithIncoherenceImpl } const int32 feature1 = m_FaceLabels[2 * i]; - const int32 feature2 = m_FaceLabels[(2 * i) + 1]; + const int32 feature2 = m_FaceLabels[2 * i + 1]; if(feature1 > 0 && feature2 > 0 && m_FeaturePhases[feature1] == m_FeaturePhases[feature2]) { - const uint32 crystalStructure = m_CrystalStructures[m_FeaturePhases[feature1]]; // Feature1 was arbitrarily selected the feature phase index is identical + const uint32 crystalStructure = m_CrystalStructures[m_FeaturePhases[feature1]]; if(crystalStructure != ebsdlib::CrystalStructure::Cubic_High && crystalStructure != ebsdlib::CrystalStructure::Cubic_Low) { continue; } - // Avg Quats is stored Vector Scalar but the Quaternion Constructor is Scalar-Vector - const Eigen::Quaterniond q1(m_AvgQuats[(feature1 * 4) + 3], m_AvgQuats[feature1 * 4], m_AvgQuats[(feature1 * 4) + 1], m_AvgQuats[(feature1 * 4) + 2]); // W X Y Z - const Eigen::Quaterniond q2(m_AvgQuats[(feature2 * 4) + 3], m_AvgQuats[feature2 * 4], m_AvgQuats[(feature2 * 4) + 1], m_AvgQuats[(feature2 * 4) + 2]); // W X Y Z + const Eigen::Quaterniond q1(m_AvgQuats[(feature1 * 4) + 3], m_AvgQuats[feature1 * 4], m_AvgQuats[(feature1 * 4) + 1], m_AvgQuats[(feature1 * 4) + 2]); + const Eigen::Quaterniond q2(m_AvgQuats[(feature2 * 4) + 3], m_AvgQuats[feature2 * 4], m_AvgQuats[(feature2 * 4) + 1], m_AvgQuats[(feature2 * 4) + 2]); const Matrix3x3 orientationMatrix = q1.matrix().transpose(); - const Eigen::Vector3d normals{m_FaceNormals[3 * i], m_FaceNormals[(3 * i) + 1], m_FaceNormals[(3 * i) + 2]}; + const Eigen::Vector3d normals{m_FaceNormals[3 * i], m_FaceNormals[3 * i + 1], m_FaceNormals[3 * i + 2]}; const Eigen::Vector3d xstl_norm = normals.transpose() * orientationMatrix; if(normals.hasNaN()) @@ -217,8 +212,8 @@ class CalculateTwinBoundaryWithIncoherenceImpl if(minIncoherence.has_value()) { - m_TwinBoundaries->setValue(i, true); - m_TwinBoundaryIncoherence[i] = static_cast(minIncoherence.value()); + m_TwinBoundariesOut[i] = 1; + m_TwinBoundaryIncoherenceOut[i] = static_cast(minIncoherence.value()); } } } @@ -232,34 +227,34 @@ class CalculateTwinBoundaryWithIncoherenceImpl private: float32 m_AxisTol; float32 m_AngTol; - const Int32AbstractDataStore& m_FaceLabels; - const Float64AbstractDataStore& m_FaceNormals; - const Float32AbstractDataStore& m_AvgQuats; - const Int32AbstractDataStore& m_FeaturePhases; - const UInt32AbstractDataStore& m_CrystalStructures; - std::unique_ptr& m_TwinBoundaries; - Float32AbstractDataStore& m_TwinBoundaryIncoherence; + const std::vector& m_FaceLabels; + const std::vector& m_FaceNormals; + const std::vector& m_AvgQuats; + const std::vector& m_FeaturePhases; + const std::vector& m_CrystalStructures; + std::vector& m_TwinBoundariesOut; + std::vector& m_TwinBoundaryIncoherenceOut; const std::atomic_bool& m_ShouldCancel; std::atomic_bool& m_HasNaN; std::vector m_OrientationOps; }; /** - * @brief The CalculateTwinBoundaryImpl class implements a threaded algorithm that determines whether a boundary is twin related. - * The calculations are performed on a surface mesh. + * @brief Parallel worker for twin boundaries without incoherence. + * All arrays are locally cached vectors — zero OOC virtual dispatch in the hot loop. */ class CalculateTwinBoundaryImpl { public: - CalculateTwinBoundaryImpl(float32 angtol, float32 axistol, const Int32AbstractDataStore& faceLabels, const Float32AbstractDataStore& avgQuats, const Int32AbstractDataStore& featurePhases, - const UInt32AbstractDataStore& crystalStructures, std::unique_ptr& twinBoundaries, const std::atomic_bool& shouldCancel) + CalculateTwinBoundaryImpl(float32 angtol, float32 axistol, const std::vector& faceLabels, const std::vector& avgQuats, const std::vector& featurePhases, + const std::vector& crystalStructures, std::vector& twinBoundariesOut, const std::atomic_bool& shouldCancel) : m_AxisTol(axistol) , m_AngTol(angtol) , m_FaceLabels(faceLabels) , m_AvgQuats(avgQuats) , m_FeaturePhases(featurePhases) , m_CrystalStructures(crystalStructures) - , m_TwinBoundaries(twinBoundaries) + , m_TwinBoundariesOut(twinBoundariesOut) , m_ShouldCancel(shouldCancel) , m_OrientationOps(ebsdlib::LaueOps::GetAllOrientationOps()) { @@ -275,19 +270,22 @@ class CalculateTwinBoundaryImpl } const int32 feature1 = m_FaceLabels[2 * i]; - const int32 feature2 = m_FaceLabels[(2 * i) + 1]; + const int32 feature2 = m_FaceLabels[2 * i + 1]; if(feature1 > 0 && feature2 > 0 && m_FeaturePhases[feature1] == m_FeaturePhases[feature2]) { - const uint32 crystalStructure = m_CrystalStructures[m_FeaturePhases[feature1]]; // Feature1 was arbitrarily selected the feature phase index is identical + const uint32 crystalStructure = m_CrystalStructures[m_FeaturePhases[feature1]]; if(crystalStructure != ebsdlib::CrystalStructure::Cubic_High && crystalStructure != ebsdlib::CrystalStructure::Cubic_Low) { continue; } - // Avg Quats is stored Vector Scalar but the Quaternion Constructor is Scalar-Vector - const Eigen::Quaterniond q1(m_AvgQuats[(feature1 * 4) + 3], m_AvgQuats[feature1 * 4], m_AvgQuats[(feature1 * 4) + 1], m_AvgQuats[(feature1 * 4) + 2]); // W X Y Z - const Eigen::Quaterniond q2(m_AvgQuats[(feature2 * 4) + 3], m_AvgQuats[feature2 * 4], m_AvgQuats[(feature2 * 4) + 1], m_AvgQuats[(feature2 * 4) + 2]); // W X Y Z - m_TwinBoundaries->setValue(i, IsTwinBoundary(q1, q2, m_OrientationOps, crystalStructure, m_AngTol, m_AxisTol)); + const Eigen::Quaterniond q1(m_AvgQuats[(feature1 * 4) + 3], m_AvgQuats[feature1 * 4], m_AvgQuats[(feature1 * 4) + 1], m_AvgQuats[(feature1 * 4) + 2]); + const Eigen::Quaterniond q2(m_AvgQuats[(feature2 * 4) + 3], m_AvgQuats[feature2 * 4], m_AvgQuats[(feature2 * 4) + 1], m_AvgQuats[(feature2 * 4) + 2]); + + if(IsTwinBoundary(q1, q2, m_OrientationOps, crystalStructure, m_AngTol, m_AxisTol)) + { + m_TwinBoundariesOut[i] = 1; + } } } } @@ -300,11 +298,11 @@ class CalculateTwinBoundaryImpl private: float32 m_AxisTol; float32 m_AngTol; - const Int32AbstractDataStore& m_FaceLabels; - const Float32AbstractDataStore& m_AvgQuats; - const Int32AbstractDataStore& m_FeaturePhases; - const UInt32AbstractDataStore& m_CrystalStructures; - std::unique_ptr& m_TwinBoundaries; + const std::vector& m_FaceLabels; + const std::vector& m_AvgQuats; + const std::vector& m_FeaturePhases; + const std::vector& m_CrystalStructures; + std::vector& m_TwinBoundariesOut; const std::atomic_bool& m_ShouldCancel; std::vector m_OrientationOps; }; @@ -332,16 +330,22 @@ const std::atomic_bool& ComputeTwinBoundaries::getCancel() // ----------------------------------------------------------------------------- Result<> ComputeTwinBoundaries::operator()() { - const auto& crystalStructures = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); + // ------------------------------------------------------------------------- + // Cache ensemble-level crystalStructures locally (tiny array) + // ------------------------------------------------------------------------- + const auto& crystalStructuresStore = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); + const usize numCrystalStructures = crystalStructuresStore.getNumberOfTuples(); + std::vector crystalStructures(numCrystalStructures); + crystalStructuresStore.copyIntoBuffer(0, nonstd::span(crystalStructures.data(), numCrystalStructures)); bool allPhasesCubic = true; bool noPhasesCubic = true; - for(usize i = 1; i < crystalStructures.size(); ++i) + for(usize i = 1; i < numCrystalStructures; ++i) { const auto crystalStructureType = crystalStructures[i]; - const bool isHex = crystalStructureType == ebsdlib::CrystalStructure::Cubic_High || crystalStructureType == ebsdlib::CrystalStructure::Cubic_Low; - allPhasesCubic = allPhasesCubic && isHex; - noPhasesCubic = noPhasesCubic && !isHex; + const bool isCubic = crystalStructureType == ebsdlib::CrystalStructure::Cubic_High || crystalStructureType == ebsdlib::CrystalStructure::Cubic_Low; + allPhasesCubic = allPhasesCubic && isCubic; + noPhasesCubic = noPhasesCubic && !isCubic; } if(noPhasesCubic) @@ -355,44 +359,96 @@ Result<> ComputeTwinBoundaries::operator()() result.warnings().push_back({-93211, "Finding the twin boundaries requires Cubic-Low m-3 or Cubic-High m-3m type crystal structures. Calculations for non Cubic phases will be skipped."}); } - const auto& faceLabels = m_DataStructure.getDataAs(m_InputValues->FaceLabelsArrayPath)->getDataStoreRef(); - const auto& avgQuats = m_DataStructure.getDataAs(m_InputValues->AvgQuatsArrayPath)->getDataStoreRef(); - const auto& featurePhases = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); + // ------------------------------------------------------------------------- + // Cache feature-level arrays locally (O(features) — thousands, not millions) + // ------------------------------------------------------------------------- + const auto& featurePhasesStore = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); + const usize numFeatures = featurePhasesStore.getNumberOfTuples(); + std::vector featurePhases(numFeatures); + featurePhasesStore.copyIntoBuffer(0, nonstd::span(featurePhases.data(), numFeatures)); - std::unique_ptr twinBoundaries; - try + const auto& avgQuatsStore = m_DataStructure.getDataAs(m_InputValues->AvgQuatsArrayPath)->getDataStoreRef(); + std::vector avgQuats(numFeatures * 4); + avgQuatsStore.copyIntoBuffer(0, nonstd::span(avgQuats.data(), numFeatures * 4)); + + // ------------------------------------------------------------------------- + // Cache face-level arrays locally (O(faces) — scales with surface area, not volume) + // ------------------------------------------------------------------------- + const auto& faceLabelsStore = m_DataStructure.getDataAs(m_InputValues->FaceLabelsArrayPath)->getDataStoreRef(); + const usize numFaces = faceLabelsStore.getNumberOfTuples(); + + std::vector faceLabels(numFaces * 2); + faceLabelsStore.copyIntoBuffer(0, nonstd::span(faceLabels.data(), numFaces * 2)); + + std::vector faceNormals; + if(m_InputValues->FindCoherence) { - twinBoundaries = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->TwinBoundariesArrayPath); - } catch(const std::out_of_range& exception) + const auto& faceNormalsStore = m_DataStructure.getDataAs(m_InputValues->FaceNormalsArrayPath)->getDataStoreRef(); + faceNormals.resize(numFaces * 3); + faceNormalsStore.copyIntoBuffer(0, nonstd::span(faceNormals.data(), numFaces * 3)); + } + + // ------------------------------------------------------------------------- + // Output buffers — parallel workers write to these, then bulk-copy to stores + // ------------------------------------------------------------------------- + std::vector twinBoundariesOut(numFaces, 0); + std::vector twinBoundaryIncoherenceOut; + if(m_InputValues->FindCoherence) { - // This really should NOT be happening as the path was verified during preflight BUT we may be calling this from - // somewhere else that is NOT going through the normal nx::core::IFilter API of Preflight and Execute - return MakeErrorResult(-93212, fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->TwinBoundariesArrayPath.toString())); + twinBoundaryIncoherenceOut.resize(numFaces, 180.0f); } const float32 angtol = m_InputValues->AngleTolerance; const float32 axistol = m_InputValues->AxisTolerance * Constants::k_PiF / 180.0f; + // ------------------------------------------------------------------------- + // Parallel execution — all data access is on local vectors, zero OOC dispatch + // ------------------------------------------------------------------------- ParallelDataAlgorithm dataAlg; - dataAlg.setRange(0, faceLabels.getNumberOfTuples()); + dataAlg.setRange(0, numFaces); + + std::atomic_bool hasNaN = false; if(m_InputValues->FindCoherence) { - std::atomic_bool hasNaN = false; - const auto& faceNormals = m_DataStructure.getDataAs(m_InputValues->FaceNormalsArrayPath)->getDataStoreRef(); - auto& twinBoundaryIncoherence = m_DataStructure.getDataAs(m_InputValues->TwinBoundaryIncoherenceArrayPath)->getDataStoreRef(); - twinBoundaryIncoherence.fill(180.0f); // For backwards compatibility - dataAlg.execute(CalculateTwinBoundaryWithIncoherenceImpl(angtol, axistol, faceLabels, faceNormals, avgQuats, featurePhases, crystalStructures, twinBoundaries, twinBoundaryIncoherence, + dataAlg.execute(CalculateTwinBoundaryWithIncoherenceImpl(angtol, axistol, faceLabels, faceNormals, avgQuats, featurePhases, crystalStructures, twinBoundariesOut, twinBoundaryIncoherenceOut, m_ShouldCancel, hasNaN)); + } + else + { + dataAlg.execute(CalculateTwinBoundaryImpl(angtol, axistol, faceLabels, avgQuats, featurePhases, crystalStructures, twinBoundariesOut, m_ShouldCancel)); + } + + // ------------------------------------------------------------------------- + // Write results back to DataStores via bulk I/O + // ------------------------------------------------------------------------- + std::unique_ptr twinBoundaries; + try + { + twinBoundaries = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->TwinBoundariesArrayPath); + } catch(const std::out_of_range& exception) + { + return MakeErrorResult(-93212, fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->TwinBoundariesArrayPath.toString())); + } - if(hasNaN.load()) + // TwinBoundaries is a MaskCompare — must write per-element (no bulk API) + for(usize i = 0; i < numFaces; i++) + { + if(twinBoundariesOut[i]) { - return MakeWarningVoidResult(-93213, fmt::format("NaNs were detected in the normals array ({}). These values were marked false.", m_InputValues->FaceNormalsArrayPath.toString())); + twinBoundaries->setValue(i, true); } } - else + + if(m_InputValues->FindCoherence) { - dataAlg.execute(CalculateTwinBoundaryImpl(angtol, axistol, faceLabels, avgQuats, featurePhases, crystalStructures, twinBoundaries, m_ShouldCancel)); + auto& incoherenceStore = m_DataStructure.getDataAs(m_InputValues->TwinBoundaryIncoherenceArrayPath)->getDataStoreRef(); + incoherenceStore.copyFromBuffer(0, nonstd::span(twinBoundaryIncoherenceOut.data(), numFaces)); } - return {}; + if(m_InputValues->FindCoherence && hasNaN.load()) + { + return MakeWarningVoidResult(-93213, fmt::format("NaNs were detected in the normals array ({}). These values were marked false.", m_InputValues->FaceNormalsArrayPath.toString())); + } + + return result; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp index 9dde1105f0..c8f7d4d489 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp @@ -11,7 +11,11 @@ #include #include +#include + +#include #include +#include #include #ifndef _MSC_VER @@ -159,22 +163,37 @@ struct StereographicCheck } \ void operator()(const Range& r) const \ { \ - InputType inputInstance; \ - size_t inNumComps = m_Input.getNumberOfComponents(); \ - size_t outNumComps = m_Output.getNumberOfComponents(); \ - for(size_t i = r.min(); i < r.max(); ++i) \ + static constexpr usize k_ChunkSize = 4096; \ + const usize inNumComps = m_Input.getNumberOfComponents(); \ + const usize outNumComps = m_Output.getNumberOfComponents(); \ + const usize totalTuples = r.max() - r.min(); \ + const usize maxChunkTuples = std::min(k_ChunkSize, totalTuples); \ + auto inBuffer = std::make_unique(maxChunkTuples * inNumComps); \ + auto outBuffer = std::make_unique(maxChunkTuples * outNumComps); \ + usize tupleIdx = r.min(); \ + while(tupleIdx < r.max()) \ { \ - size_t inOffset = i * inNumComps; \ - size_t outOffset = i * outNumComps; \ - for(size_t c = 0; c < inNumComps; c++) \ - { \ - inputInstance[c] = m_Input[inOffset + c]; \ - } \ - OutputType outputInstance = inputInstance.to##TO_REP(); \ - for(size_t c = 0; c < outNumComps; c++) \ + const usize chunkTuples = std::min(k_ChunkSize, r.max() - tupleIdx); \ + const usize inElemCount = chunkTuples * inNumComps; \ + const usize outElemCount = chunkTuples * outNumComps; \ + m_Input.copyIntoBuffer(tupleIdx* inNumComps, nonstd::span(inBuffer.get(), inElemCount)); \ + InputType inputInstance; \ + for(usize t = 0; t < chunkTuples; ++t) \ { \ - m_Output[outOffset + c] = outputInstance[c]; \ + const usize inOff = t * inNumComps; \ + const usize outOff = t * outNumComps; \ + for(usize c = 0; c < inNumComps; ++c) \ + { \ + inputInstance[c] = inBuffer[inOff + c]; \ + } \ + OutputType outputInstance = inputInstance.to##TO_REP(); \ + for(usize c = 0; c < outNumComps; ++c) \ + { \ + outBuffer[outOff + c] = outputInstance[c]; \ + } \ } \ + m_Output.copyFromBuffer(tupleIdx* outNumComps, nonstd::span(outBuffer.get(), outElemCount)); \ + tupleIdx += chunkTuples; \ } \ } \ \ @@ -214,16 +233,16 @@ Result<> ConvertOrientations::operator()() DataPath outputDataPath = m_InputValues->InputOrientationArrayPath.replaceName(m_InputValues->OutputOrientationArrayName); auto inputArray = m_DataStructure.getDataRefAs(m_InputValues->InputOrientationArrayPath); auto outputArray = m_DataStructure.getDataRefAs(outputDataPath); - size_t totalPoints = inputArray.getNumberOfTuples(); - - const ValidateInputDataFunctionType euCheck = EulerCheck(); - const ValidateInputDataFunctionType omCheck = OrientationMatrixCheck(); - const ValidateInputDataFunctionType quCheck = QuaternionCheck(); - const ValidateInputDataFunctionType axCheck = AxisAngleCheck(); - const ValidateInputDataFunctionType roCheck = RodriguesCheck(); - const ValidateInputDataFunctionType hoCheck = HomochoricCheck(); - const ValidateInputDataFunctionType cuCheck = CubochoricCheck(); - const ValidateInputDataFunctionType stCheck = StereographicCheck(); + usize totalPoints = inputArray.getNumberOfTuples(); + + const ValidateInputDataFunctionType euCheck = EulerCheck(); + const ValidateInputDataFunctionType omCheck = OrientationMatrixCheck(); + const ValidateInputDataFunctionType quCheck = QuaternionCheck(); + const ValidateInputDataFunctionType axCheck = AxisAngleCheck(); + const ValidateInputDataFunctionType roCheck = RodriguesCheck(); + const ValidateInputDataFunctionType hoCheck = HomochoricCheck(); + const ValidateInputDataFunctionType cuCheck = CubochoricCheck(); + const ValidateInputDataFunctionType stCheck = StereographicCheck(); // Allow data-based parallelization ParallelDataAlgorithm parallelAlgorithm; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.cpp index 000b1e28b3..d2151135d2 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.cpp @@ -2,6 +2,10 @@ #include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +#include +#include using namespace nx::core; @@ -17,6 +21,30 @@ EBSDSegmentFeatures::EBSDSegmentFeatures(DataStructure& dataStructure, const IFi // ----------------------------------------------------------------------------- EBSDSegmentFeatures::~EBSDSegmentFeatures() noexcept = default; +// ----------------------------------------------------------------------------- +// Segments an EBSD dataset into crystallographic features (grains) by flood- +// filling contiguous voxels whose crystal orientations are within a user- +// specified misorientation tolerance. Two voxels are grouped into the same +// feature only if they share the same phase and their misorientation (computed +// via the appropriate LaueOps symmetry operator) is below the threshold. +// +// Algorithm dispatch: +// - In-core data -> execute() : classic depth-first-search (DFS) flood fill +// - Out-of-core -> executeCCL() : connected-component labeling that streams +// data slice-by-slice to limit memory usage +// The choice is made by checking IsOutOfCore() on the FeatureIds array (i.e., +// whether the backing DataStore lives on disk) or if ForceOocAlgorithm() is +// set (used for testing). +// +// Post-processing after either algorithm: +// 1. Validate that at least one feature was found (error if not). +// 2. Resize the Feature AttributeMatrix to (m_FoundFeatures + 1) tuples so +// that all per-feature arrays (Active, etc.) have the correct size. +// Index 0 is reserved as an invalid/background feature. +// 3. Initialize the Active array: fill with 1 (active), then set index 0 +// to 0 to mark it as the reserved background slot. +// 4. Optionally randomize FeatureIds so that spatially adjacent grains get +// non-sequential IDs, improving visual contrast in color-mapped renders. // ----------------------------------------------------------------------------- Result<> EBSDSegmentFeatures::operator()() { @@ -43,8 +71,21 @@ Result<> EBSDSegmentFeatures::operator()() m_FeatureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); m_FeatureIdsArray->fill(0); // initialize the output array with zeros - // Run the segmentation algorithm - execute(gridGeom); + // Dispatch between DFS (in-core) and CCL (OOC) algorithms + if(IsOutOfCore(*m_FeatureIdsArray) || ForceOocAlgorithm()) + { + SizeVec3 udims = gridGeom->getDimensions(); + allocateSliceBuffers(static_cast(udims[0]), static_cast(udims[1])); + + auto& featureIdsStore = m_FeatureIdsArray->getDataStoreRef(); + executeCCL(gridGeom, featureIdsStore); + + deallocateSliceBuffers(); + } + else + { + execute(gridGeom); + } // Sanity check the result. if(this->m_FoundFeatures < 1) { @@ -73,7 +114,23 @@ Result<> EBSDSegmentFeatures::operator()() } // ----------------------------------------------------------------------------- -int64_t EBSDSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const +// Finds the next unassigned voxel that can serve as the seed for a new feature. +// The scan is a simple linear walk starting from `nextSeed`, which is the index +// immediately after the last seed found. This avoids rescanning already-assigned +// voxels at the beginning of the array. +// +// A voxel is eligible to become a seed when all three conditions are met: +// 1. featureId == 0 : the voxel has not yet been assigned to any feature. +// 2. Passes the mask: if masking is enabled, the voxel must be flagged as +// "good" (e.g., not a bad scan point). +// 3. cellPhase > 0 : the voxel belongs to a real crystallographic phase +// (phase 0 is reserved for unindexed/background points). +// +// When a valid seed is found, its featureId is immediately set to `gnum` +// (the new feature number) so that subsequent calls will skip it. +// Returns the linear index of the seed, or -1 if no more seeds exist. +// ----------------------------------------------------------------------------- +int64 EBSDSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const { nx::core::DataArray::store_type* featureIds = m_FeatureIdsArray->getDataStore(); usize totalPoints = featureIds->getNumberOfTuples(); @@ -108,6 +165,26 @@ int64_t EBSDSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const return seed; } +// ----------------------------------------------------------------------------- +// Determines whether a neighboring voxel should be merged into the current +// feature during the DFS flood fill (execute() path). This is NOT used by +// the CCL path, which calls areNeighborsSimilar() instead. +// +// The method checks three conditions before grouping: +// 1. The neighbor's featureId must be 0 (unassigned). +// 2. The neighbor must pass the mask (if masking is enabled). +// 3. The neighbor must be crystallographically similar to the reference voxel. +// +// Similarity check (misorientation): +// - Look up the Laue class for both voxels from their phase -> crystal +// structure mapping. If either Laue class is out of range (>= number of +// known symmetry operators, e.g., phase == 999), bail out immediately. +// - Extract the quaternion orientations (4 floats per voxel) for both points. +// - If both voxels share the same phase, compute the misorientation angle via +// LaueOps::calculateMisorientation(), which returns an axis-angle pair. +// The angle (w, in radians) accounts for crystal symmetry equivalences. +// - If w < MisorientationTolerance, the voxels are considered part of the +// same grain. The neighbor's featureId is set to `gnum` as a side effect. // ----------------------------------------------------------------------------- bool EBSDSegmentFeatures::determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const { @@ -116,8 +193,8 @@ bool EBSDSegmentFeatures::determineGrouping(int64 referencePoint, int64 neighbor // Get the phases for each voxel nx::core::AbstractDataStore* cellPhases = m_CellPhases->getDataStore(); - int32_t laueClass1 = (*m_CrystalStructures)[(*cellPhases)[referencePoint]]; - int32_t laueClass2 = (*m_CrystalStructures)[(*cellPhases)[neighborPoint]]; + int32 laueClass1 = (*m_CrystalStructures)[(*cellPhases)[referencePoint]]; + int32 laueClass2 = (*m_CrystalStructures)[(*cellPhases)[neighborPoint]]; // If either of the phases is 999 then we bail out now. if(laueClass1 >= m_OrientationOps.size() || laueClass2 >= m_OrientationOps.size()) { @@ -134,14 +211,14 @@ bool EBSDSegmentFeatures::determineGrouping(int64 referencePoint, int64 neighbor if(featureIds[neighborPoint] == 0 && (m_GoodVoxelsArray == nullptr || neighborPointIsGood)) { - float w = std::numeric_limits::max(); + float32 w = std::numeric_limits::max(); const ebsdlib::QuatD q1(currentQuatPtr[referencePoint * 4], currentQuatPtr[referencePoint * 4 + 1], currentQuatPtr[referencePoint * 4 + 2], currentQuatPtr[referencePoint * 4 + 3]); const ebsdlib::QuatD q2(currentQuatPtr[neighborPoint * 4], currentQuatPtr[neighborPoint * 4 + 1], currentQuatPtr[neighborPoint * 4 + 2], currentQuatPtr[neighborPoint * 4 + 3]); if((*cellPhases)[referencePoint] == (*cellPhases)[neighborPoint]) { ebsdlib::AxisAngleDType axisAngle = m_OrientationOps[laueClass1]->calculateMisorientation(q1, q2); - w = static_cast(axisAngle[3]); + w = static_cast(axisAngle[3]); } if(w < m_InputValues->MisorientationTolerance) { @@ -152,3 +229,304 @@ bool EBSDSegmentFeatures::determineGrouping(int64 referencePoint, int64 neighbor return group; } + +// ----------------------------------------------------------------------------- +// Checks whether a single voxel is eligible for segmentation (used by the CCL +// path in executeCCL()). A voxel is valid if it passes the mask and has a +// crystallographic phase > 0. +// +// Slice buffer fast path: +// When m_UseSliceBuffers is true (OOC mode), the method first checks whether +// the voxel's Z-slice is currently loaded in the rolling 2-slot buffer. The +// slot is determined by (iz % 2). If the voxel's slice matches the buffered +// slice index, the mask and phase values are read directly from the in-memory +// m_MaskBuffer and m_PhaseBuffer arrays, avoiding an on-disk I/O round-trip. +// +// OOC fallback: +// If slice buffers are not active, or if the voxel's slice is not currently +// buffered (which can happen during Phase 1b of CCL when periodic boundary +// merging accesses non-adjacent slices), the method falls back to direct +// array access through the DataStore, which may trigger on-disk I/O for +// out-of-core data. +// ----------------------------------------------------------------------------- +bool EBSDSegmentFeatures::isValidVoxel(int64 point) const +{ + if(m_UseSliceBuffers) + { + const int64 iz = point / m_BufSliceSize; + const int slot = static_cast(iz % 2); + if(m_BufferedSliceZ[slot] == iz) + { + const usize sliceSize = static_cast(m_BufSliceSize); + const usize off = static_cast(slot) * sliceSize + static_cast(point - iz * m_BufSliceSize); + if(m_InputValues->UseMask && m_MaskBuffer[off] == 0) + { + return false; + } + if(m_PhaseBuffer[off] <= 0) + { + return false; + } + return true; + } + } + + // OOC fallback + if(m_InputValues->UseMask && !m_GoodVoxelsArray->isTrue(point)) + { + return false; + } + AbstractDataStore& cellPhases = m_CellPhases->getDataStoreRef(); + if(cellPhases[point] <= 0) + { + return false; + } + return true; +} + +// ----------------------------------------------------------------------------- +// Determines whether two neighboring voxels are crystallographically similar +// enough to belong to the same feature. Used exclusively by the CCL path +// (executeCCL()), whereas the DFS path uses determineGrouping() instead. +// +// Slice buffer fast path: +// When both voxels' Z-slices are present in the rolling 2-slot buffer, all +// data is read from the in-memory buffers (m_QuatBuffer, m_PhaseBuffer, +// m_MaskBuffer). The buffer offset for each point is computed as: +// slot * sliceSize + (point - iz * sliceSize) +// For quaternions, an additional x4 factor accounts for the 4 components +// per voxel. The method then: +// 1. Checks point2's mask validity. +// 2. Checks that point2's phase > 0 and both phases match. +// 3. Looks up the Laue class and verifies it is in range. +// 4. Constructs QuatD objects from the buffered quaternion components. +// 5. Computes misorientation via LaueOps::calculateMisorientation(). +// 6. Returns true if the misorientation angle < MisorientationTolerance. +// +// OOC fallback: +// If either voxel's slice is not buffered (e.g., during Phase 1b periodic +// merge), falls back to direct DataStore access: validates point2 via +// isValidVoxel(), checks phase equality, then computes misorientation from +// the full quaternion and phase arrays on disk. +// ----------------------------------------------------------------------------- +bool EBSDSegmentFeatures::areNeighborsSimilar(int64 point1, int64 point2) const +{ + if(m_UseSliceBuffers) + { + const int64 iz1 = point1 / m_BufSliceSize; + const int slot1 = static_cast(iz1 % 2); + const int64 iz2 = point2 / m_BufSliceSize; + const int slot2 = static_cast(iz2 % 2); + + if(m_BufferedSliceZ[slot1] == iz1 && m_BufferedSliceZ[slot2] == iz2) + { + const usize sliceSize = static_cast(m_BufSliceSize); + const usize off1 = static_cast(slot1) * sliceSize + static_cast(point1 - iz1 * m_BufSliceSize); + const usize off2 = static_cast(slot2) * sliceSize + static_cast(point2 - iz2 * m_BufSliceSize); + + // Check point2 validity + if(m_InputValues->UseMask && m_MaskBuffer[off2] == 0) + { + return false; + } + const int32 phase1 = m_PhaseBuffer[off1]; + const int32 phase2 = m_PhaseBuffer[off2]; + if(phase2 <= 0) + { + return false; + } + if(phase1 != phase2) + { + return false; + } + + int32 laueClass = static_cast(m_CrystalStructuresCache[static_cast(phase1)]); + if(static_cast(laueClass) >= m_OrientationOps.size()) + { + return false; + } + + const usize q1Base = static_cast(slot1) * sliceSize * 4 + static_cast(point1 - iz1 * m_BufSliceSize) * 4; + const usize q2Base = static_cast(slot2) * sliceSize * 4 + static_cast(point2 - iz2 * m_BufSliceSize) * 4; + + const ebsdlib::QuatD q1(m_QuatBuffer[q1Base], m_QuatBuffer[q1Base + 1], m_QuatBuffer[q1Base + 2], m_QuatBuffer[q1Base + 3]); + const ebsdlib::QuatD q2(m_QuatBuffer[q2Base], m_QuatBuffer[q2Base + 1], m_QuatBuffer[q2Base + 2], m_QuatBuffer[q2Base + 3]); + + ebsdlib::AxisAngleDType axisAngle = m_OrientationOps[laueClass]->calculateMisorientation(q1, q2); + float32 w = static_cast(axisAngle[3]); + + return w < m_InputValues->MisorientationTolerance; + } + } + + // OOC fallback (original code) + if(!isValidVoxel(point2)) + { + return false; + } + + AbstractDataStore& cellPhases = m_CellPhases->getDataStoreRef(); + + if(cellPhases[point1] != cellPhases[point2]) + { + return false; + } + + int32 laueClass = (*m_CrystalStructures)[cellPhases[point1]]; + if(static_cast(laueClass) >= m_OrientationOps.size()) + { + return false; + } + + Float32Array& quats = *m_QuatsArray; + const ebsdlib::QuatD q1(quats[point1 * 4], quats[point1 * 4 + 1], quats[point1 * 4 + 2], quats[point1 * 4 + 3]); + const ebsdlib::QuatD q2(quats[point2 * 4], quats[point2 * 4 + 1], quats[point2 * 4 + 2], quats[point2 * 4 + 3]); + + ebsdlib::AxisAngleDType axisAngle = m_OrientationOps[laueClass]->calculateMisorientation(q1, q2); + float32 w = static_cast(axisAngle[3]); + + return w < m_InputValues->MisorientationTolerance; +} + +// ----------------------------------------------------------------------------- +// Allocates the rolling 2-slot slice buffers used by the CCL (OOC) algorithm. +// Called once at the start of the OOC branch in operator(), before executeCCL(). +// +// Each slot holds one full XY slice (dimX * dimY voxels). Two slots are needed +// because the CCL algorithm compares the current slice (iz) with the previous +// slice (iz-1), so both must be in memory simultaneously. +// +// Buffers allocated: +// - m_QuatBuffer : 2 * sliceSize * 4 floats (quaternion: 4 components/voxel) +// - m_PhaseBuffer : 2 * sliceSize int32 values (one phase ID per voxel) +// - m_MaskBuffer : 2 * sliceSize uint8 values (one mask flag per voxel) +// +// Both m_BufferedSliceZ slots are initialized to -1 (no slice loaded). +// m_UseSliceBuffers is set to true so that isValidVoxel() and +// areNeighborsSimilar() will use the fast buffer path. +// ----------------------------------------------------------------------------- +void EBSDSegmentFeatures::allocateSliceBuffers(int64 dimX, int64 dimY) +{ + m_BufSliceSize = dimX * dimY; + const usize sliceSize = static_cast(m_BufSliceSize); + m_QuatBuffer.resize(2 * sliceSize * 4); + m_PhaseBuffer.resize(2 * sliceSize); + m_MaskBuffer.resize(2 * sliceSize); + m_BufferedSliceZ[0] = -1; + m_BufferedSliceZ[1] = -1; + m_UseSliceBuffers = true; + + // Cache crystal structures locally to avoid per-voxel OOC access. + // This array is tiny (one entry per phase) but gets accessed ~24M times + // during the CCL inner loop; going through an OOC DataStore each time is + // the dominant bottleneck. + const usize numPhases = m_CrystalStructures->getNumberOfTuples(); + m_CrystalStructuresCache.resize(numPhases); + m_CrystalStructures->getDataStoreRef().copyIntoBuffer(0, nonstd::span(m_CrystalStructuresCache.data(), numPhases)); +} + +// ----------------------------------------------------------------------------- +// Releases the slice buffers after executeCCL() completes, freeing the memory +// back to the system. Called in the OOC branch of operator() after the CCL +// algorithm finishes. Resets m_UseSliceBuffers to false and both +// m_BufferedSliceZ slots to -1. The vectors are replaced with default- +// constructed (empty) instances to guarantee memory deallocation. +// ----------------------------------------------------------------------------- +void EBSDSegmentFeatures::deallocateSliceBuffers() +{ + m_UseSliceBuffers = false; + m_QuatBuffer = std::vector(); + m_PhaseBuffer = std::vector(); + m_MaskBuffer = std::vector(); + m_CrystalStructuresCache = std::vector(); + m_BufferedSliceZ[0] = -1; + m_BufferedSliceZ[1] = -1; +} + +// ----------------------------------------------------------------------------- +// Pre-loads voxel data for a single Z-slice into the rolling 2-slot buffer, +// called by executeCCL() before processing each slice. +// +// Rolling buffer design: +// The target slot is determined by (iz % 2), so even slices go to slot 0 and +// odd slices go to slot 1. Because the CCL algorithm processes slices in +// order (0, 1, 2, ...), at any given slice iz the previous slice (iz-1) is +// always in the other slot, keeping both the current and previous slice data +// available in memory. +// +// Sentinel behavior: +// If iz < 0, slice buffering is disabled (m_UseSliceBuffers = false). The +// CCL algorithm passes iz = -1 after completing the slice-by-slice sweep to +// signal that subsequent calls (e.g., during Phase 1b periodic boundary +// merging) should use direct DataStore access instead of the buffers. +// +// Skip-if-already-loaded: +// If m_BufferedSliceZ[slot] == iz, the data for this slice is already in the +// buffer (e.g., from a previous prepareForSlice call), so the method returns +// immediately without re-reading. +// +// Data loaded per slice: +// - Quaternions (4 float32 per voxel) into m_QuatBuffer +// - Phase IDs (1 int32 per voxel) into m_PhaseBuffer +// - Mask flags (1 uint8 per voxel) into m_MaskBuffer; if masking is disabled, +// all mask values are set to 1 (valid) +// ----------------------------------------------------------------------------- +void EBSDSegmentFeatures::prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) +{ + if(iz < 0) + { + m_UseSliceBuffers = false; + return; + } + + if(!m_UseSliceBuffers) + { + return; + } + + const int slot = static_cast(iz % 2); + if(m_BufferedSliceZ[slot] == iz) + { + return; + } + + const usize sliceSize = static_cast(m_BufSliceSize); + const usize slotOffset = static_cast(slot) * sliceSize; + const usize quatSlotOffset = slotOffset * 4; + const int64 baseIndex = iz * m_BufSliceSize; + + // Bulk-read quaternions (4 components per voxel) for this slice + AbstractDataStore& quatStore = m_QuatsArray->getDataStoreRef(); + quatStore.copyIntoBuffer(static_cast(baseIndex) * 4, nonstd::span(m_QuatBuffer.data() + quatSlotOffset, sliceSize * 4)); + + // Bulk-read phase IDs for this slice + AbstractDataStore& phaseStore = m_CellPhases->getDataStoreRef(); + phaseStore.copyIntoBuffer(static_cast(baseIndex), nonstd::span(m_PhaseBuffer.data() + slotOffset, sliceSize)); + + // Bulk-read mask flags for this slice + if(m_InputValues->UseMask && m_GoodVoxelsArray != nullptr) + { + auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + if(maskArray.getDataType() == DataType::uint8) + { + auto& typedStore = maskArray.getIDataStoreRefAs>(); + typedStore.copyIntoBuffer(static_cast(baseIndex), nonstd::span(m_MaskBuffer.data() + slotOffset, sliceSize)); + } + else if(maskArray.getDataType() == DataType::boolean) + { + auto& typedStore = maskArray.getIDataStoreRefAs>(); + auto boolBuf = std::make_unique(sliceSize); + typedStore.copyIntoBuffer(static_cast(baseIndex), nonstd::span(boolBuf.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + m_MaskBuffer[slotOffset + i] = boolBuf[i] ? 1 : 0; + } + } + } + else + { + std::fill(m_MaskBuffer.begin() + slotOffset, m_MaskBuffer.begin() + slotOffset + sliceSize, static_cast(1)); + } + + m_BufferedSliceZ[slot] = iz; +} diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp index d1db6caada..8409e3ba5e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp @@ -56,26 +56,25 @@ class ORIENTATIONANALYSIS_EXPORT EBSDSegmentFeatures : public SegmentFeatures Result<> operator()(); protected: + int64 getSeed(int32 gnum, int64 nextSeed) const override; + bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; + /** - * @brief - * @param data - * @param args - * @param gnum - * @param nextSeed - * @return int64 + * @brief Checks whether a voxel can participate in EBSD segmentation based on mask and phase. + * @param point Linear voxel index. + * @return true if the voxel passes mask and phase checks. */ - int64_t getSeed(int32 gnum, int64 nextSeed) const override; + bool isValidVoxel(int64 point) const override; /** - * @brief - * @param data - * @param args - * @param referencepoint - * @param neighborpoint - * @param gnum - * @return bool + * @brief Determines whether two neighboring voxels belong to the same EBSD segment. + * @param point1 First voxel index. + * @param point2 Second (neighbor) voxel index. + * @return true if both voxels share the same phase and their misorientation is within tolerance. */ - bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; + bool areNeighborsSimilar(int64 point1, int64 point2) const override; + + void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) override; private: const EBSDSegmentFeaturesInputValues* m_InputValues = nullptr; @@ -87,6 +86,20 @@ class ORIENTATIONANALYSIS_EXPORT EBSDSegmentFeatures : public SegmentFeatures FeatureIdsArrayType* m_FeatureIdsArray = nullptr; std::vector m_OrientationOps; + + void allocateSliceBuffers(int64 dimX, int64 dimY); + void deallocateSliceBuffers(); + + // Rolling 2-slot input buffers for OOC optimization. + // Pre-loading input data into these avoids per-element OOC overhead + // during neighbor comparisons in the CCL algorithm. + std::vector m_QuatBuffer; + std::vector m_PhaseBuffer; + std::vector m_MaskBuffer; + std::vector m_CrystalStructuresCache; + int64 m_BufSliceSize = 0; + int64 m_BufferedSliceZ[2] = {-1, -1}; + bool m_UseSliceBuffers = false; }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp index 2700ac0b82..e82c5a5a46 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp @@ -183,10 +183,23 @@ Result<> MergeTwins::operator()() * There is code later on to ensure that only m3m Laue class is used. */ auto& laueClasses = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); - auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); - auto& cellParentIds = m_DataStructure.getDataAs(m_InputValues->CellParentIdsArrayPath)->getDataStoreRef(); - cellParentIds.fill(-1); + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + auto& cellParentIdsStore = m_DataStructure.getDataAs(m_InputValues->CellParentIdsArrayPath)->getDataStoreRef(); auto& featureParentIds = m_DataStructure.getDataAs(m_InputValues->FeatureParentIdsArrayPath)->getDataStoreRef(); + + usize totalPoints = cellParentIdsStore.getNumberOfTuples(); + + // Chunked fill of cellParentIds for OOC efficiency + { + constexpr usize k_FillChunk = 65536; + std::vector fillBuf(k_FillChunk, -1); + for(usize offset = 0; offset < totalPoints; offset += k_FillChunk) + { + usize count = std::min(k_FillChunk, totalPoints - offset); + cellParentIdsStore.copyFromBuffer(offset, nonstd::span(fillBuf.data(), count)); + } + } + featureParentIds.fill(-1); for(usize i = 1; i < laueClasses.getSize(); i++) @@ -217,21 +230,39 @@ Result<> MergeTwins::operator()() result, ConvertResult(MakeErrorResult(-23501, "The number of grouped Features was 0 or 1 which means no grouped Features were detected. A grouping value may be set too high"))); } - // Update data arrays. + // Cache feature-level featureParentIds locally for chunked voxel loop + const usize numFeatures = featureParentIds.getNumberOfTuples(); + std::vector featureParentIdsCache(numFeatures); + featureParentIds.copyIntoBuffer(0, nonstd::span(featureParentIdsCache.data(), numFeatures)); + + // Chunked bulk I/O for voxel-level parent ID assignment int32 numParents = 0; - usize totalPoints = featureIds.getNumberOfTuples(); - for(usize k = 0; k < totalPoints; k++) { - if(m_ShouldCancel) - { - return {}; - } + constexpr usize k_ChunkSize = 65536; + std::vector featureIdsBuf(k_ChunkSize); + std::vector cellParentIdsBuf(k_ChunkSize); - int32 featureName = featureIds[k]; - cellParentIds[k] = featureParentIds[featureName]; - if(featureParentIds[featureName] > numParents) + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) { - numParents = featureParentIds[featureName]; + if(m_ShouldCancel) + { + return {}; + } + + usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(featureIdsBuf.data(), count)); + + for(usize i = 0; i < count; i++) + { + int32 featureName = featureIdsBuf[i]; + cellParentIdsBuf[i] = featureParentIdsCache[featureName]; + if(featureParentIdsCache[featureName] > numParents) + { + numParents = featureParentIdsCache[featureName]; + } + } + + cellParentIdsStore.copyFromBuffer(offset, nonstd::span(cellParentIdsBuf.data(), count)); } } numParents += 1; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.cpp index c2a49646e4..4e70155be5 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.cpp @@ -7,59 +7,12 @@ #include "simplnx/Utilities/DataGroupUtilities.hpp" #include "simplnx/Utilities/MessageHelper.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" -#include "simplnx/Utilities/ParallelTaskAlgorithm.hpp" - -#ifdef SIMPLNX_ENABLE_MULTICORE -#define RUN_TASK g->run -#else -#define RUN_TASK -#endif +#include "simplnx/Utilities/SliceBufferedTransfer.hpp" #include using namespace nx::core; -class NeighborOrientationCorrelationTransferDataImpl -{ -public: - NeighborOrientationCorrelationTransferDataImpl() = delete; - NeighborOrientationCorrelationTransferDataImpl(const NeighborOrientationCorrelationTransferDataImpl&) = default; - - NeighborOrientationCorrelationTransferDataImpl(MessageHelper& messageHelper, size_t totalPoints, const std::vector& bestNeighbor, std::shared_ptr dataArrayPtr) - : m_MessageHelper(messageHelper) - , m_TotalPoints(totalPoints) - , m_BestNeighbor(bestNeighbor) - , m_DataArrayPtr(dataArrayPtr) - { - } - NeighborOrientationCorrelationTransferDataImpl(NeighborOrientationCorrelationTransferDataImpl&&) = default; // Move Constructor Not Implemented - NeighborOrientationCorrelationTransferDataImpl& operator=(const NeighborOrientationCorrelationTransferDataImpl&) = delete; // Copy Assignment Not Implemented - NeighborOrientationCorrelationTransferDataImpl& operator=(NeighborOrientationCorrelationTransferDataImpl&&) = delete; // Move Assignment Not Implemented - - ~NeighborOrientationCorrelationTransferDataImpl() = default; - - void operator()() const - { - ThrottledMessenger throttledMessenger = m_MessageHelper.createThrottledMessenger(); - std::string arrayName = m_DataArrayPtr->getName(); - for(size_t i = 0; i < m_TotalPoints; i++) - { - throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Processing {}: {:.2f}% completed", arrayName, CalculatePercentComplete(i, m_TotalPoints)); }); - int64 neighbor = m_BestNeighbor[i]; - if(neighbor != -1) - { - m_DataArrayPtr->copyTuple(neighbor, i); - } - } - } - -private: - MessageHelper& m_MessageHelper; - size_t m_TotalPoints = 0; - std::vector m_BestNeighbor; - std::shared_ptr m_DataArrayPtr; -}; - // ----------------------------------------------------------------------------- NeighborOrientationCorrelation::NeighborOrientationCorrelation(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, NeighborOrientationCorrelationInputValues* inputValues) @@ -76,18 +29,26 @@ NeighborOrientationCorrelation::~NeighborOrientationCorrelation() noexcept = def // ----------------------------------------------------------------------------- Result<> NeighborOrientationCorrelation::operator()() { - size_t progress = 0; - size_t totalProgress = 0; - std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - const auto& confidenceIndex = m_DataStructure.getDataRefAs(m_InputValues->ConfidenceIndexArrayPath); - const auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); - const auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); - const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); - size_t totalPoints = confidenceIndex.getNumberOfTuples(); + auto& confidenceIndex = m_DataStructure.getDataRefAs(m_InputValues->ConfidenceIndexArrayPath); + auto& cellPhases = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath); + auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); + const auto& crystalStructuresArray = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + + // Cache ensemble-level arrays locally to avoid per-element virtual dispatch in hot loops + const auto& crystalStructuresStore = crystalStructuresArray.getDataStoreRef(); + const usize numPhases = crystalStructuresStore.getNumberOfTuples(); + std::vector crystalStructures(numPhases); + crystalStructuresStore.copyIntoBuffer(0, nonstd::span(crystalStructures.data(), numPhases)); - float misorientationToleranceR = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; + const auto& ciStore = confidenceIndex.getDataStoreRef(); + const auto& phaseStore = cellPhases.getDataStoreRef(); + const auto& quatStore = quats.getDataStoreRef(); + + usize totalPoints = confidenceIndex.getNumberOfTuples(); + + float32 misorientationToleranceR = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeomPath); SizeVec3 udims = imageGeom.getDimensions(); @@ -98,108 +59,182 @@ Result<> NeighborOrientationCorrelation::operator()() static_cast(udims[2]), }; - int32 best = 0; - int64 neighborPoint2 = 0; - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - std::vector neighborDiffCount(totalPoints, 0); - std::vector neighborSimCount(6, 0); - std::vector bestNeighbor(totalPoints, -1); + std::array neighborSimCount = {}; const int32 startLevel = 6; MessageHelper messageHelper(m_MessageHandler); - ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); + // Z-slice buffering: read 3 adjacent Z-slices of the most-accessed arrays + // into local memory to eliminate OOC chunk thrashing. The algorithm accesses + // each voxel's 6 face neighbors, requiring data from z-1, z, and z+1 slices. + // By buffering these slices, all neighbor lookups become local memory reads. + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + + // Rolling window: slot 0 = z-1, slot 1 = z (current), slot 2 = z+1 + std::array, 3> quatSlices; + std::array, 3> phaseSlices; + std::vector ciSlice(sliceSize); + + for(auto& qs : quatSlices) + { + qs.resize(sliceSize * 4); + } + for(auto& ps : phaseSlices) + { + ps.resize(sliceSize); + } + + // Bulk-read a Z-slice using copyIntoBuffer for OOC efficiency + auto readQuatSlice = [&](int64 z, usize slot) { + const usize zOffset = static_cast(z) * sliceSize * 4; + quatStore.copyIntoBuffer(zOffset, nonstd::span(quatSlices[slot].data(), sliceSize * 4)); + }; + + auto readPhaseSlice = [&](int64 z, usize slot) { + const usize zOffset = static_cast(z) * sliceSize; + phaseStore.copyIntoBuffer(zOffset, nonstd::span(phaseSlices[slot].data(), sliceSize)); + }; + + auto readCISlice = [&](int64 z) { + const usize zOffset = static_cast(z) * sliceSize; + ciStore.copyIntoBuffer(zOffset, nonstd::span(ciSlice.data(), sliceSize)); + }; + + // Per-slice best neighbor marks (replaces O(totalPoints) bestNeighbor array) + std::vector sliceBestNeighbor(sliceSize, -1); + const usize dimZ = static_cast(dims[2]); + std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->ConfidenceIndexArrayPath, m_InputValues->IgnoredDataArrayPaths); + for(int32 currentLevel = startLevel; currentLevel > m_InputValues->Level; currentLevel--) { - for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) + usize processedVoxels = 0; + + // Initialize rolling window: load z=0 into slot 1, z=1 into slot 2 + readQuatSlice(0, 1); + readPhaseSlice(0, 1); + if(dims[2] > 1) { - throttledMessenger.sendThrottledMessage([&]() { - return fmt::format("Level '{}' of '{}' || Processing Data {:.2f}% completed", (startLevel - currentLevel) + 1, startLevel - m_InputValues->Level, - CalculatePercentComplete(voxelIndex, totalPoints)); - }); + readQuatSlice(1, 2); + readPhaseSlice(1, 2); + } - if(m_ShouldCancel) + for(int64 zIdx = 0; zIdx < dims[2] && !m_ShouldCancel; zIdx++) + { + // Advance rolling window for z > 0 + if(zIdx > 0) { - break; + std::swap(quatSlices[0], quatSlices[1]); + std::swap(quatSlices[1], quatSlices[2]); + std::swap(phaseSlices[0], phaseSlices[1]); + std::swap(phaseSlices[1], phaseSlices[2]); + if(zIdx + 1 < dims[2]) + { + readQuatSlice(zIdx + 1, 2); + readPhaseSlice(zIdx + 1, 2); + } } - if(confidenceIndex[voxelIndex] < m_InputValues->MinConfidence) + readCISlice(zIdx); + + for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { - int64 xIdx = voxelIndex % dims[0]; - int64 yIdx = (voxelIndex / dims[0]) % dims[1]; - int64 zIdx = voxelIndex / (dims[0] * dims[1]); - // Loop over the 6 face neighbors of the voxel - std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(const auto& faceIndexJ : faceNeighborInternalIdx) + for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { - if(!isValidFaceNeighbor[faceIndexJ]) - { - continue; - } - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndexJ]; + int64 voxelIndex = xIdx + yIdx * dims[0] + zIdx * static_cast(sliceSize); + usize inSlice = static_cast(yIdx * dims[0] + xIdx); - uint32 laueClass = crystalStructures[cellPhases[voxelIndex]]; - ebsdlib::QuatD quat1(quats[voxelIndex * 4], quats[voxelIndex * 4 + 1], quats[voxelIndex * 4 + 2], quats[voxelIndex * 4 + 3]); - ebsdlib::QuatD quat2(quats[neighborPoint * 4], quats[neighborPoint * 4 + 1], quats[neighborPoint * 4 + 2], quats[neighborPoint * 4 + 3]); - ebsdlib::AxisAngleDType axisAngle(0.0, 0.0, 0.0, std::numeric_limits::max()); - if(cellPhases[voxelIndex] == cellPhases[neighborPoint] && cellPhases[voxelIndex] > 0) - { - axisAngle = orientationOps[laueClass]->calculateMisorientation(quat1, quat2); - } - if(axisAngle[3] > misorientationToleranceR) + if(processedVoxels % 10000 == 0) { - neighborDiffCount[voxelIndex]++; + throttledMessenger.sendThrottledMessage([&]() { + return fmt::format("Level '{}' of '{}' || Processing Data {:.2f}% completed", (startLevel - currentLevel) + 1, startLevel - m_InputValues->Level, + CalculatePercentComplete(processedVoxels, totalPoints)); + }); } - isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(size_t faceIndexK = faceIndexJ + 1; faceIndexK < k_FaceNeighborCount; faceIndexK++) + if(ciSlice[inSlice] < m_InputValues->MinConfidence) { - if(!isValidFaceNeighbor[faceIndexK]) + std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + + // Pre-read all valid neighbor quats and phases into local arrays. + // Neighbor buffer slots: 0=-Z, 1=-Y(same z), 2=-X(same z), 3=+X(same z), 4=+Y(same z), 5=+Z + // slot mapping: -Z→0, same-z→1, +Z→2 + constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + const std::array neighborBufX = {xIdx, xIdx, xIdx - 1, xIdx + 1, xIdx, xIdx}; + const std::array neighborBufY = {yIdx, yIdx - 1, yIdx, yIdx, yIdx + 1, yIdx}; + + std::array nQuats; + std::array nPhases = {}; + + for(usize f = 0; f < k_FaceNeighborCount; f++) { - continue; + if(isValidFaceNeighbor[f]) + { + usize nIdx = static_cast(neighborBufY[f] * dims[0] + neighborBufX[f]); + usize nIdx4 = nIdx * 4; + nPhases[f] = phaseSlices[k_NeighborSlot[f]][nIdx]; + nQuats[f] = + ebsdlib::QuatD(quatSlices[k_NeighborSlot[f]][nIdx4], quatSlices[k_NeighborSlot[f]][nIdx4 + 1], quatSlices[k_NeighborSlot[f]][nIdx4 + 2], quatSlices[k_NeighborSlot[f]][nIdx4 + 3]); + } } - neighborPoint2 = voxelIndex + neighborVoxelIndexOffsets[faceIndexK]; - laueClass = crystalStructures[cellPhases[neighborPoint2]]; - quat1 = ebsdlib::QuatD(quats[neighborPoint2 * 4], quats[neighborPoint2 * 4 + 1], quats[neighborPoint2 * 4 + 2], quats[neighborPoint2 * 4 + 3]); - quat2 = ebsdlib::QuatD(quats[neighborPoint * 4], quats[neighborPoint * 4 + 1], quats[neighborPoint * 4 + 2], quats[neighborPoint * 4 + 3]); - axisAngle = ebsdlib::AxisAngleDType(0.0, 0.0, 0.0, std::numeric_limits::max()); - if(cellPhases[neighborPoint2] == cellPhases[neighborPoint] && cellPhases[neighborPoint2] > 0) + // Compute neighbor-neighbor similarity counts + neighborSimCount.fill(0); + + for(usize faceIndexJ = 0; faceIndexJ < k_FaceNeighborCount; faceIndexJ++) { - axisAngle = orientationOps[laueClass]->calculateMisorientation(quat1, quat2); + if(!isValidFaceNeighbor[faceIndexJ]) + { + continue; + } + + for(usize faceIndexK = faceIndexJ + 1; faceIndexK < k_FaceNeighborCount; faceIndexK++) + { + if(!isValidFaceNeighbor[faceIndexK]) + { + continue; + } + + if(nPhases[faceIndexK] == nPhases[faceIndexJ] && nPhases[faceIndexK] > 0) + { + uint32 laueClass = crystalStructures[nPhases[faceIndexK]]; + ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass]->calculateMisorientation(nQuats[faceIndexK], nQuats[faceIndexJ]); + if(axisAngle[3] < misorientationToleranceR) + { + neighborSimCount[faceIndexJ]++; + neighborSimCount[faceIndexK]++; + } + } + } } - if(axisAngle[3] < misorientationToleranceR) + + // Find the best neighbor (last valid face with positive similarity count) + for(usize faceIndex = 0; faceIndex < k_FaceNeighborCount; faceIndex++) { - neighborSimCount[faceIndexJ]++; - neighborSimCount[faceIndexK]++; + if(!isValidFaceNeighbor[faceIndex]) + { + continue; + } + + if(neighborSimCount[faceIndex] > 0) + { + sliceBestNeighbor[inSlice] = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + } } } - } - - // Loop over the 6 face neighbors of the voxel - isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(const auto& faceIndex : faceNeighborInternalIdx) - { - if(!isValidFaceNeighbor[faceIndex]) - { - continue; - } - best = 0; - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - - if(neighborSimCount[faceIndex] > best) - { - best = neighborSimCount[faceIndex]; - bestNeighbor[voxelIndex] = neighborPoint; - } - neighborSimCount[faceIndex] = 0; + processedVoxels++; } } + + // Transfer this Z-slice immediately (bestNeighbor only marks the current voxel) + for(const auto& dataArrayPtr : voxelArrays) + { + SliceBufferedTransferOneZ(*dataArrayPtr, sliceBestNeighbor, sliceSize, static_cast(zIdx), dimZ); + } + std::fill(sliceBestNeighbor.begin(), sliceBestNeighbor.end(), -1); } if(m_ShouldCancel) @@ -207,17 +242,6 @@ Result<> NeighborOrientationCorrelation::operator()() return {}; } - // Build up a list of the DataArrays that we are going to operate on. - std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->ConfidenceIndexArrayPath, m_InputValues->IgnoredDataArrayPaths); - // The idea for this parallel section is to parallelize over each Data Array that - // will need it's data adjusted. This should go faster than before by about 2x. - // Better speed up could be achieved if we had better data locality. - ParallelTaskAlgorithm parallelTask; - for(const auto& dataArrayPtr : voxelArrays) - { - parallelTask.execute(NeighborOrientationCorrelationTransferDataImpl(messageHelper, totalPoints, bestNeighbor, dataArrayPtr)); - } - currentLevel = currentLevel - 1; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.hpp index 0b21553a6d..ed95965e33 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/NeighborOrientationCorrelation.hpp @@ -10,27 +10,65 @@ namespace nx::core { +/** + * @struct NeighborOrientationCorrelationInputValues + * @brief Holds all user-supplied parameters for the NeighborOrientationCorrelation algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelationInputValues { - DataPath ImageGeomPath; - float32 MinConfidence; - float32 MisorientationTolerance; - int32 Level; - DataPath ConfidenceIndexArrayPath; - DataPath CellPhasesArrayPath; - DataPath QuatsArrayPath; - DataPath CrystalStructuresArrayPath; - MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; + DataPath ImageGeomPath; ///< Path to the ImageGeom that defines the voxel grid dimensions + float32 MinConfidence = 0.0f; ///< Cells with confidence index below this value are candidates for replacement + float32 MisorientationTolerance = 0.0f; ///< Angular tolerance (degrees) for comparing neighbor orientations + int32 Level = 0; ///< Minimum neighbor agreement count required to replace a cell (cleanup level) + DataPath ConfidenceIndexArrayPath; ///< Path to the float32 confidence index array + DataPath CellPhasesArrayPath; ///< Path to the int32 cell phases array + DataPath QuatsArrayPath; ///< Path to the float32 quaternion array (4 components per tuple) + DataPath CrystalStructuresArrayPath; ///< Path to the uint32 crystal structures ensemble array + MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; ///< Data arrays excluded from the neighbor-copy transfer step }; /** - * @class + * @class NeighborOrientationCorrelation + * @brief Corrects low-confidence EBSD voxels by replacing their cell data with + * data from the most orientation-correlated face neighbor. + * + * The algorithm iterates through multiple "cleanup levels" (from 6 down to the + * user-specified Level). At each level, every voxel whose confidence index is + * below MinConfidence is examined. For that voxel, the 6 face neighbors are + * compared pairwise: two neighbors "agree" if they share the same nonzero phase + * and their misorientation is within MisorientationTolerance. Each neighbor + * accumulates a similarity count (how many other neighbors agree with it). The + * neighbor with the highest agreement is chosen as the replacement source. + * + * ## Z-Slice Buffering (Out-of-Core Optimization) + * + * To avoid random-access thrashing of out-of-core (OOC) compressed chunk stores, + * the algorithm maintains a rolling window of 3 adjacent Z-slices for the + * quaternion and phase arrays, plus 1 Z-slice for the confidence index. At each + * Z-step, the window advances by swapping buffer slots and reading only the new + * z+1 slice. All neighbor lookups then read from these local buffers instead of + * the backing DataArray, eliminating repeated chunk decompressions. + * + * After identifying the best neighbor for every low-confidence voxel in a level, + * all cell-level DataArrays (except ignored ones) are updated in parallel using + * ParallelTaskAlgorithm, copying tuple data from each best neighbor. */ class ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelation { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays + * @param mesgHandler Handler for sending progress messages to the UI + * @param shouldCancel Atomic flag checked between iterations to support cancellation + * @param inputValues User-supplied parameters controlling the algorithm behavior + */ NeighborOrientationCorrelation(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, NeighborOrientationCorrelationInputValues* inputValues); + + /** + * @brief Default destructor. + */ ~NeighborOrientationCorrelation() noexcept; NeighborOrientationCorrelation(const NeighborOrientationCorrelation&) = delete; @@ -38,6 +76,10 @@ class ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelation NeighborOrientationCorrelation& operator=(const NeighborOrientationCorrelation&) = delete; NeighborOrientationCorrelation& operator=(NeighborOrientationCorrelation&&) noexcept = delete; + /** + * @brief Executes the neighbor orientation correlation algorithm. + * @return Result<> indicating success or any errors encountered during execution + */ Result<> operator()(); private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp index 4b34d4c179..5541992edb 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp @@ -8,6 +8,8 @@ #include +#include + using namespace nx::core; using FloatVec3Type = std::vector; @@ -103,13 +105,11 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const { const DataPath CellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName); - std::vector cDims = {1}; - const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->DataContainerName); - const size_t totalCells = imageGeom.getNumberOfCells(); + const usize totalCells = imageGeom.getNumberOfCells(); // Prepare the Cell Attribute Matrix with the correct number of tuples based on the total Cells being read from the file. - std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; + std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; // Adjust the values of the 'phase' data to correct for invalid values and assign the read Phase Data into the actual DataArray { @@ -118,15 +118,16 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const return; } auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::Phases)); - int* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::PhaseData)); - for(size_t i = 0; i < totalCells; i++) + auto* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::PhaseData)); + // Validate phases in-place (original code also modifies phasePtr) + for(usize i = 0; i < totalCells; i++) { if(phasePtr[i] < 1) { phasePtr[i] = 1; } - targetArray[i] = phasePtr[i]; } + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(phasePtr, totalCells)); } // Condense the Euler Angles from 3 separate arrays into a single 1x3 array @@ -135,17 +136,25 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const { return; } - const auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Phi1)); - const auto* fComp1 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Phi)); - const auto* fComp2 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Phi2)); - cDims[0] = 3; + const auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Phi1)); + const auto* fComp1 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Phi)); + const auto* fComp2 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Phi2)); auto& cellEulerAngles = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::EulerAngles)); - for(size_t i = 0; i < totalCells; i++) + auto& eulerStore = cellEulerAngles.getDataStoreRef(); + + constexpr usize k_ChunkSize = 65536; + std::vector eulerBuf(k_ChunkSize * 3); + for(usize offset = 0; offset < totalCells; offset += k_ChunkSize) { - cellEulerAngles[3 * i] = fComp0[i]; - cellEulerAngles[3 * i + 1] = fComp1[i]; - cellEulerAngles[3 * i + 2] = fComp2[i]; + usize count = std::min(k_ChunkSize, totalCells - offset); + for(usize i = 0; i < count; i++) + { + eulerBuf[3 * i] = fComp0[offset + i]; + eulerBuf[3 * i + 1] = fComp1[offset + i]; + eulerBuf[3 * i + 2] = fComp2[offset + i]; + } + eulerStore.copyFromBuffer(offset * 3, nonstd::span(eulerBuf.data(), count * 3)); } } @@ -153,40 +162,40 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const { return; } - cDims[0] = 1; + { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::ImageQuality)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::ImageQuality)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ImageQuality)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::ConfidenceIndex)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::ConfidenceIndex)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ConfidenceIndex)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::SEMSignal)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::SEMSignal)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::SEMSignal)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Fit)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::Fit)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::Fit)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::XPosition)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::XPosition)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::XPosition)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::YPosition)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::YPosition)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::YPosition)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp index 6c4f58e434..244bccaa95 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp @@ -7,6 +7,8 @@ #include #include +#include + using namespace nx::core; using FloatVec3Type = std::vector; @@ -101,94 +103,106 @@ void ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const { const DataPath cellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName); const DataPath cellEnsembleAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellEnsembleAttributeMatrixName); - std::vector cDims = {1}; const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->DataContainerName); - const size_t totalCells = imageGeom.getNumberOfCells(); + const usize totalCells = imageGeom.getNumberOfCells(); // Prepare the Cell Attribute Matrix with the correct number of tuples based on the total Cells being read from the file. - std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; + std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; // Copy the Phase Array { auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::Phases)); - int* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Phase)); - for(size_t i = 0; i < totalCells; i++) - { - targetArray[i] = phasePtr[i]; - } + auto* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Phase)); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(phasePtr, totalCells)); } // Condense the Euler Angles from 3 separate arrays into a single 1x3 array { auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); - auto& cellPhases = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::Phases)); + const auto* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Phase)); + + // Cache ensemble-level crystal structures locally + const auto& csStore = crystalStructures.getDataStoreRef(); + const usize numPhases = csStore.getNumberOfTuples(); + std::vector csCache(numPhases); + csStore.copyIntoBuffer(0, nonstd::span(csCache.data(), numPhases)); - const auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler1)); - const auto* fComp1 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler2)); - const auto* fComp2 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler3)); - cDims[0] = 3; + const auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler1)); + const auto* fComp1 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler2)); + const auto* fComp2 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Euler3)); auto& cellEulerAngles = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::EulerAngles)); - for(size_t i = 0; i < totalCells; i++) + auto& eulerStore = cellEulerAngles.getDataStoreRef(); + + constexpr usize k_ChunkSize = 65536; + std::vector eulerBuf(k_ChunkSize * 3); + for(usize offset = 0; offset < totalCells; offset += k_ChunkSize) { - cellEulerAngles[3 * i] = fComp0[i]; - cellEulerAngles[3 * i + 1] = fComp1[i]; - cellEulerAngles[3 * i + 2] = fComp2[i]; - if(crystalStructures[cellPhases[i]] == ebsdlib::CrystalStructure::Hexagonal_High && m_InputValues->EdaxHexagonalAlignment) - { - cellEulerAngles[3 * i + 2] = cellEulerAngles[3 * i + 2] + 30.0F; // See the documentation for this correction factor - } - // Now convert to radians if requested by the user - if(m_InputValues->DegreesToRadians) + usize count = std::min(k_ChunkSize, totalCells - offset); + for(usize i = 0; i < count; i++) { - cellEulerAngles[3 * i] = cellEulerAngles[3 * i] * ebsdlib::constants::k_PiOver180F; - cellEulerAngles[3 * i + 1] = cellEulerAngles[3 * i + 1] * ebsdlib::constants::k_PiOver180F; - cellEulerAngles[3 * i + 2] = cellEulerAngles[3 * i + 2] * ebsdlib::constants::k_PiOver180F; + float32 e0 = fComp0[offset + i]; + float32 e1 = fComp1[offset + i]; + float32 e2 = fComp2[offset + i]; + if(csCache[phasePtr[offset + i]] == ebsdlib::CrystalStructure::Hexagonal_High && m_InputValues->EdaxHexagonalAlignment) + { + e2 = e2 + 30.0F; // See the documentation for this correction factor + } + // Now convert to radians if requested by the user + if(m_InputValues->DegreesToRadians) + { + e0 = e0 * ebsdlib::constants::k_PiOver180F; + e1 = e1 * ebsdlib::constants::k_PiOver180F; + e2 = e2 * ebsdlib::constants::k_PiOver180F; + } + eulerBuf[3 * i] = e0; + eulerBuf[3 * i + 1] = e1; + eulerBuf[3 * i + 2] = e2; } + eulerStore.copyFromBuffer(offset * 3, nonstd::span(eulerBuf.data(), count * 3)); } } - cDims[0] = 1; { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Bands)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Bands)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Bands)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Error)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Error)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Error)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::MAD)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::MAD)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::MAD)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::BC)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::BC)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::BC)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::BS)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::BS)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::BS)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::X)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::X)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::X)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } { - auto* fComp0 = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Y)); + auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Y)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Y)); - std::copy(fComp0, fComp0 + totalCells, targetArray.begin()); + targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(srcPtr, totalCells)); } } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp index 05085246ad..f1d6a76057 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp @@ -5,6 +5,7 @@ #include "simplnx/Common/Numbers.hpp" #include "simplnx/Common/StringLiteral.hpp" #include "simplnx/Common/TypeTraits.hpp" +#include "simplnx/Common/Types.hpp" #include "simplnx/Core/Application.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/DataStructure/StringArray.hpp" @@ -20,6 +21,8 @@ #include #include +using namespace nx::core; + namespace { // Parameter Keys @@ -31,7 +34,7 @@ constexpr nx::core::StringLiteral k_RotateSliceBySlice_Key = "rotate_slice_by_sl constexpr nx::core::StringLiteral k_RemoveOriginalGeometry_Key = "remove_original_geometry"; // constexpr nx::core::StringLiteral k_RotatedGeometryName = ".RotatedGeometry"; -enum class RotationRepresentation : uint64_t +enum class RotationRepresentation : uint64 { AxisAngle = 0, RotationMatrix = 1 @@ -59,7 +62,7 @@ nx::core::Result<> LoadInfo(const nx::core::ReadH5EbsdInputValues* mInputValues, } // Resize the Ensemble Attribute Matrix to be the correct number of phases. - std::vector tDims = {phases.size() + 1}; + ShapeType tDims = {phases.size() + 1}; nx::core::DataPath cellEnsembleMatrixPath = mInputValues->cellEnsembleMatrixPath; @@ -87,12 +90,12 @@ nx::core::Result<> LoadInfo(const nx::core::ReadH5EbsdInputValues* mInputValues, latticData[4] = 0.0f; latticData[5] = 0.0f; - for(size_t i = 0; i < phases.size(); i++) + for(usize i = 0; i < phases.size(); i++) { - int32_t phaseID = phases[i]->getPhaseIndex(); + int32 phaseID = phases[i]->getPhaseIndex(); xtalData[phaseID] = phases[i]->determineOrientationOpsIndex(); matNameData[phaseID] = phases[i]->getMaterialName(); - std::vector latticeConstant = phases[i]->getLatticeConstants(); + std::vector latticeConstant = phases[i]->getLatticeConstants(); latticData[phaseID * 6ULL] = latticeConstant[0]; latticData[phaseID * 6ULL + 1] = latticeConstant[1]; @@ -107,7 +110,7 @@ nx::core::Result<> LoadInfo(const nx::core::ReadH5EbsdInputValues* mInputValues, template void CopyData(nx::core::DataStructure& dataStructure, H5EbsdReaderType* ebsdReader, const std::vector& arrayNames, std::set selectedArrayNames, - const nx::core::DataPath& cellAttributeMatrixPath, size_t totalPoints) + const nx::core::DataPath& cellAttributeMatrixPath, usize totalPoints) { using DataArrayType = nx::core::DataArray; for(const auto& arrayName : arrayNames) @@ -115,12 +118,9 @@ void CopyData(nx::core::DataStructure& dataStructure, H5EbsdReaderType* ebsdRead if(selectedArrayNames.find(arrayName) != selectedArrayNames.end()) { T* source = reinterpret_cast(ebsdReader->getPointerByName(arrayName)); - nx::core::DataPath dataPath = cellAttributeMatrixPath.createChildPath(arrayName); // get the data from the DataStructure + nx::core::DataPath dataPath = cellAttributeMatrixPath.createChildPath(arrayName); auto& destination = dataStructure.getDataRefAs(dataPath); - for(size_t tupleIndex = 0; tupleIndex < totalPoints; tupleIndex++) - { - destination[tupleIndex] = source[tupleIndex]; - } + destination.getDataStoreRef().copyFromBuffer(0, nonstd::span(source, totalPoints * destination.getNumberOfComponents())); } } } @@ -139,10 +139,10 @@ void CopyData(nx::core::DataStructure& dataStructure, H5EbsdReaderType* ebsdRead */ template nx::core::Result<> LoadEbsdData(const nx::core::ReadH5EbsdInputValues* mInputValues, nx::core::DataStructure& dataStructure, const std::vector& eulerNames, - const nx::core::IFilter::MessageHandler& mMessageHandler, std::set selectedArrayNames, const std::array& dcDims, + const nx::core::IFilter::MessageHandler& mMessageHandler, std::set selectedArrayNames, const std::array& dcDims, const std::vector& floatArrayNames, const std::vector& intArrayNames) { - int32_t err = 0; + int32 err = 0; std::shared_ptr ebsdReader = std::dynamic_pointer_cast(H5EbsdReaderType::New()); if(nullptr == ebsdReader) { @@ -170,7 +170,7 @@ nx::core::Result<> LoadEbsdData(const nx::core::ReadH5EbsdInputValues* mInputVal // Initialize all the arrays with some default values mMessageHandler(nx::core::IFilter::Message{nx::core::IFilter::Message::Type::Info, fmt::format("Reading EBSD Data from file {}", mInputValues->inputFilePath)}); - uint32_t mRefFrameZDir = ebsdReader->getStackingOrder(); + uint32 mRefFrameZDir = ebsdReader->getStackingOrder(); ebsdReader->setSliceStart(mInputValues->startSlice); ebsdReader->setSliceEnd(mInputValues->endSlice); @@ -185,7 +185,7 @@ nx::core::Result<> LoadEbsdData(const nx::core::ReadH5EbsdInputValues* mInputVal nx::core::DataPath geometryPath = mInputValues->dataContainerPath; nx::core::DataPath cellAttributeMatrixPath = mInputValues->cellAttributeMatrixPath; - size_t totalPoints = dcDims[0] * dcDims[1] * dcDims[2]; + usize totalPoints = dcDims[0] * dcDims[1] * dcDims[2]; // Get the Crystal Structure data which should have already been read from the file and copied to the array nx::core::DataPath cellEnsembleMatrixPath = mInputValues->cellEnsembleMatrixPath; @@ -193,55 +193,70 @@ nx::core::Result<> LoadEbsdData(const nx::core::ReadH5EbsdInputValues* mInputVal auto& xtalData = dataStructure.getDataRefAs(xtalDataPath); // Copy the Phase Values from the EBSDReader to the DataStructure - auto* phasePtr = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[3])); // get the phase data from the EbsdReader + auto* phasePtr = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[3])); // get the phase data from the EbsdReader nx::core::DataPath phaseDataPath = cellAttributeMatrixPath.createChildPath(ebsdlib::H5Ebsd::Phases); // get the phase data from the DataStructure nx::core::Int32Array* phaseDataArrayPtr = nullptr; if(selectedArrayNames.find(eulerNames[3]) != selectedArrayNames.end()) { phaseDataArrayPtr = dataStructure.getDataAs(phaseDataPath); - for(size_t tupleIndex = 0; tupleIndex < totalPoints; tupleIndex++) - { - (*phaseDataArrayPtr)[tupleIndex] = phasePtr[tupleIndex]; - } + phaseDataArrayPtr->getDataStoreRef().copyFromBuffer(0, nonstd::span(phasePtr, totalPoints)); } if(selectedArrayNames.find(ebsdlib::CellData::EulerAngles) != selectedArrayNames.end()) { // radian conversion = std::numbers::pi / 180.0; - auto* euler0 = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[0])); - auto* euler1 = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[1])); - auto* euler2 = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[2])); - // std::vector cDims = {3}; + auto* euler0 = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[0])); + auto* euler1 = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[1])); + auto* euler2 = reinterpret_cast(ebsdReader->getPointerByName(eulerNames[2])); + // ShapeType cDims = {3}; nx::core::DataPath eulerDataPath = cellAttributeMatrixPath.createChildPath(ebsdlib::CellData::EulerAngles); // get the Euler data from the DataStructure auto& eulerData = dataStructure.getDataRefAs(eulerDataPath); - float degToRad = 1.0f; + float32 degToRad = 1.0f; if(mInputValues->eulerRepresentation != ebsdlib::AngleRepresentation::Radians && mInputValues->useRecommendedTransform) { - degToRad = nx::core::numbers::pi_v / 180.0F; + degToRad = nx::core::numbers::pi_v / 180.0F; } - for(size_t elementIndex = 0; elementIndex < totalPoints; elementIndex++) + // Interleave 3 separate Euler arrays into [e0,e1,e2, e0,e1,e2, ...] layout using a chunked buffer + // Also apply Oxford hex correction if needed (avoids a second pass over the data) + constexpr usize k_ChunkTuples = 65536; + auto eulerBuf = std::make_unique(k_ChunkTuples * 3); + + // Cache phase data and crystal structures locally if Oxford hex correction is needed + std::unique_ptr phaseCache; + std::unique_ptr xtalCache; + bool applyHexCorrection = (manufacturer == ebsdlib::Ctf::Manufacturer && phaseDataArrayPtr != nullptr); + if(applyHexCorrection) { - eulerData[3 * elementIndex] = euler0[elementIndex] * degToRad; - eulerData[3 * elementIndex + 1] = euler1[elementIndex] * degToRad; - eulerData[3 * elementIndex + 2] = euler2[elementIndex] * degToRad; + phaseCache = std::make_unique(totalPoints); + phaseDataArrayPtr->getDataStoreRef().copyIntoBuffer(0, nonstd::span(phaseCache.get(), totalPoints)); + usize numXtal = xtalData.getSize(); + xtalCache = std::make_unique(numXtal); + xtalData.getDataStoreRef().copyIntoBuffer(0, nonstd::span(xtalCache.get(), numXtal)); } - // THIS IS ONLY TO BRING OXFORD DATA INTO THE SAME HEX REFERENCE AS EDAX HEX REFERENCE - if(manufacturer == ebsdlib::Ctf::Manufacturer && phaseDataArrayPtr != nullptr) + + auto& eulerStore = eulerData.getDataStoreRef(); + for(usize startTup = 0; startTup < totalPoints; startTup += k_ChunkTuples) { - for(size_t elementIndex = 0; elementIndex < totalPoints; elementIndex++) + usize count = std::min(k_ChunkTuples, totalPoints - startTup); + for(usize i = 0; i < count; i++) { - if(xtalData[(*phaseDataArrayPtr)[elementIndex]] == ebsdlib::CrystalStructure::Hexagonal_High) + usize srcIdx = startTup + i; + eulerBuf[i * 3] = euler0[srcIdx] * degToRad; + eulerBuf[i * 3 + 1] = euler1[srcIdx] * degToRad; + eulerBuf[i * 3 + 2] = euler2[srcIdx] * degToRad; + if(applyHexCorrection && xtalCache[phaseCache[srcIdx]] == ebsdlib::CrystalStructure::Hexagonal_High) { - eulerData[3 * elementIndex + 2] = eulerData[3 * elementIndex + 2] + (30.0F * degToRad); + eulerBuf[i * 3 + 2] += (30.0F * degToRad); } } + eulerStore.copyFromBuffer(startTup * 3, nonstd::span(eulerBuf.get(), count * 3)); } } // Copy the EBSD Data from its temp location into the final DataStructure location. - ::CopyData(dataStructure, ebsdReader.get(), floatArrayNames, selectedArrayNames, cellAttributeMatrixPath, totalPoints); + ::CopyData(dataStructure, ebsdReader.get(), floatArrayNames, selectedArrayNames, cellAttributeMatrixPath, totalPoints); ::CopyData(dataStructure, ebsdReader.get(), intArrayNames, selectedArrayNames, cellAttributeMatrixPath, totalPoints); return {}; @@ -249,8 +264,6 @@ nx::core::Result<> LoadEbsdData(const nx::core::ReadH5EbsdInputValues* mInputVal } // namespace -using namespace nx::core; - // ----------------------------------------------------------------------------- ReadH5Ebsd::ReadH5Ebsd(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadH5EbsdInputValues* inputValues) : m_DataStructure(dataStructure) @@ -274,22 +287,22 @@ Result<> ReadH5Ebsd::operator()() { return MakeErrorResult(-50000, fmt::format("Could not read H5EbsdVolumeInfo from file '{}", m_InputValues->inputFilePath)); } - std::array dims = {0, 0, 0}; - std::array res = {0.0f, 0.0f, 0.0f}; + std::array dims = {0, 0, 0}; + std::array res = {0.0f, 0.0f, 0.0f}; volumeInfoReader->getDimsAndResolution(dims[0], dims[1], dims[2], res[0], res[1], res[2]); - std::array dcDims = {static_cast(dims[0]), static_cast(dims[1]), static_cast(dims[2])}; + std::array dcDims = {static_cast(dims[0]), static_cast(dims[1]), static_cast(dims[2])}; // Now Calculate our "subvolume" of slices, ie, those start and end values that the user selected from the GUI dcDims[2] = m_InputValues->endSlice - m_InputValues->startSlice + 1; std::string manufacturer = volumeInfoReader->getManufacturer(); - std::array sampleTransAxis = volumeInfoReader->getSampleTransformationAxis(); - float sampleTransAngle = volumeInfoReader->getSampleTransformationAngle(); + std::array sampleTransAxis = volumeInfoReader->getSampleTransformationAxis(); + float32 sampleTransAngle = volumeInfoReader->getSampleTransformationAngle(); - std::array eulerTransAxis = volumeInfoReader->getEulerTransformationAxis(); - float eulerTransAngle = volumeInfoReader->getEulerTransformationAngle(); + std::array eulerTransAxis = volumeInfoReader->getEulerTransformationAxis(); + float32 eulerTransAngle = volumeInfoReader->getEulerTransformationAngle(); // This will effectively close the reader and free any memory being used volumeInfoReader = ebsdlib::H5EbsdVolumeInfo::NullPointer(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp index 7669cec89b..50ebf8af0c 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp @@ -58,27 +58,32 @@ Result<> ReadH5EspritData::copyRawEbsdData(int index) const auto* yBm = reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5Esprit::YBEAM)); auto& yBeam = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5Esprit::YBEAM)); - for(size_t i = 0; i < totalPoints; i++) + // Interleave 3 separate Euler angle arrays into 1x3 using bounded chunks { - // Condense the Euler Angles from 3 separate arrays into a single 1x3 array - eulerAngles[offset + 3 * i] = phi1[i] * degToRad; - eulerAngles[offset + 3 * i + 1] = phi[i] * degToRad; - eulerAngles[offset + 3 * i + 2] = phi2[i] * degToRad; - - mad[offset + i] = m1[i]; - - nIndexBands[offset + i] = nIndBands[i]; - - phase[offset + i] = p1[i]; - - radonBandCount[offset + i] = radBandCnt[i]; - - radonQuality[offset + i] = radQual[i]; - - xBeam[offset + i] = xBm[i]; - - yBeam[offset + i] = yBm[i]; + constexpr usize k_ChunkTuples = 65536; + std::vector eulerChunk(k_ChunkTuples * 3); + auto& eulerStore = eulerAngles.getDataStoreRef(); + for(usize chunkStart = 0; chunkStart < totalPoints; chunkStart += k_ChunkTuples) + { + const usize chunkCount = std::min(k_ChunkTuples, totalPoints - chunkStart); + for(usize i = 0; i < chunkCount; i++) + { + eulerChunk[i * 3] = phi1[chunkStart + i] * degToRad; + eulerChunk[i * 3 + 1] = phi[chunkStart + i] * degToRad; + eulerChunk[i * 3 + 2] = phi2[chunkStart + i] * degToRad; + } + eulerStore.copyFromBuffer((offset + chunkStart) * 3, nonstd::span(eulerChunk.data(), chunkCount * 3)); + } } + + // Bulk copy single-component arrays directly from HDF5 reader buffers + mad.getDataStoreRef().copyFromBuffer(offset, nonstd::span(m1, totalPoints)); + nIndexBands.getDataStoreRef().copyFromBuffer(offset, nonstd::span(nIndBands, totalPoints)); + phase.getDataStoreRef().copyFromBuffer(offset, nonstd::span(p1, totalPoints)); + radonBandCount.getDataStoreRef().copyFromBuffer(offset, nonstd::span(radBandCnt, totalPoints)); + radonQuality.getDataStoreRef().copyFromBuffer(offset, nonstd::span(radQual, totalPoints)); + xBeam.getDataStoreRef().copyFromBuffer(offset, nonstd::span(xBm, totalPoints)); + yBeam.getDataStoreRef().copyFromBuffer(offset, nonstd::span(yBm, totalPoints)); } if(m_InputValues->ReadPatternData) @@ -97,13 +102,7 @@ Result<> ReadH5EspritData::copyRawEbsdData(int index) pDimsV[1] = pDims[1]; auto& patternData = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5Esprit::RawPatterns)); const usize numComponents = patternData.getNumberOfComponents(); - for(usize i = 0; i < totalPoints; i++) - { - for(usize j = 0; j < numComponents; ++j) - { - patternData[offset + numComponents * i + j] = patternDataPtr[numComponents * i + j]; - } - } + patternData.getDataStoreRef().copyFromBuffer(offset * numComponents, nonstd::span(patternDataPtr, totalPoints * numComponents)); } } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp index 9f38ec997d..cf2e31295e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp @@ -5,8 +5,6 @@ #include "simplnx/Common/Array.hpp" #include "simplnx/Common/Numbers.hpp" #include "simplnx/DataStructure/DataArray.hpp" -#include "simplnx/Utilities/MessageHelper.hpp" -#include "simplnx/Utilities/ParallelDataAlgorithm.hpp" #include #include @@ -14,76 +12,6 @@ using namespace nx::core; -namespace -{ - -/** - * @brief The RotateEulerRefFrameImpl class implements a threaded algorithm that rotates an array of Euler - * angles about the supplied axis-angle pair. - */ -class RotateEulerRefFrameImpl -{ - -public: - RotateEulerRefFrameImpl(Float32Array& data, const FloatVec3& rotAxis, float angle, const std::atomic_bool& shouldCancel, ProgressMessageHelper& progressMessageHelper) - : m_CellEulerAngles(data) - , m_AxisAngle(rotAxis) - , m_Angle(angle) - , m_ShouldCancel(shouldCancel) - , m_ProgressMessageHelper(progressMessageHelper) - { - } - virtual ~RotateEulerRefFrameImpl() = default; - - void convert(size_t start, size_t end) const - { - ebsdlib::OrientationMatrixDType om = ebsdlib::AxisAngleDType(m_AxisAngle[0], m_AxisAngle[1], m_AxisAngle[2], m_Angle * nx::core::numbers::pi / 180.0).toOrientationMatrix(); - - OrientationUtilities::Matrix3dR rotMat = om.toEigenGMatrix(); - - ProgressMessenger progressMessenger = m_ProgressMessageHelper.createProgressMessenger(); - - usize counter = 0; - usize counterIncrement = (end - start) / 100; - // float ea1 = 0, ea2 = 0, ea3 = 0; - for(size_t i = start; i < end; i++) - { - if(m_ShouldCancel) - { - return; - } - if(counter >= counterIncrement) - { - progressMessenger.sendProgressMessage(counter); - counter = 0; - } - - om = ebsdlib::EulerDType(m_CellEulerAngles[3 * i + 0], m_CellEulerAngles[3 * i + 1], m_CellEulerAngles[3 * i + 2]).toOrientationMatrix(); - OrientationUtilities::Matrix3dR gNew = (om * rotMat).colwise().normalized(); - - ebsdlib::EulerDType eu = ebsdlib::OrientationMatrixDType(gNew.data()).toEuler(); - m_CellEulerAngles[3 * i] = eu[0]; - m_CellEulerAngles[3 * i + 1] = eu[1]; - m_CellEulerAngles[3 * i + 2] = eu[2]; - counter++; - } - progressMessenger.sendProgressMessage(counter); - } - - void operator()(const Range& range) const - { - convert(range.min(), range.max()); - } - -private: - Float32Array& m_CellEulerAngles; - FloatVec3 m_AxisAngle; - float m_Angle = 0.0F; - const std::atomic_bool& m_ShouldCancel; - ProgressMessageHelper& m_ProgressMessageHelper; -}; -} // namespace - // ----------------------------------------------------------------------------- RotateEulerRefFrame::RotateEulerRefFrame(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, RotateEulerRefFrameInputValues* inputValues) : m_DataStructure(dataStructure) @@ -104,25 +32,47 @@ Result<> RotateEulerRefFrame::operator()() return {}; } - nx::core::Float32Array& eulerAngles = m_DataStructure.getDataRefAs(m_InputValues->eulerAngleDataPath); - - size_t totalElements = eulerAngles.getNumberOfTuples(); + auto& eulerAngles = m_DataStructure.getDataRefAs(m_InputValues->eulerAngleDataPath); + auto& eulerStore = eulerAngles.getDataStoreRef(); + const usize totalTuples = eulerAngles.getNumberOfTuples(); - nx::core::FloatVec3 axis = {m_InputValues->rotationAxis[0], m_InputValues->rotationAxis[1], m_InputValues->rotationAxis[2]}; + FloatVec3 axis = {m_InputValues->rotationAxis[0], m_InputValues->rotationAxis[1], m_InputValues->rotationAxis[2]}; axis = axis.normalize(); + const float32 angle = m_InputValues->rotationAxis[3]; + + ebsdlib::OrientationMatrixDType omRot = ebsdlib::AxisAngleDType(axis[0], axis[1], axis[2], angle * nx::core::numbers::pi / 180.0).toOrientationMatrix(); + OrientationUtilities::Matrix3dR rotMat = omRot.toEigenGMatrix(); + + // Process in bounded chunks: read → rotate → write back + constexpr usize k_ChunkTuples = 65536; + std::vector buf(k_ChunkTuples * 3); + + for(usize startTup = 0; startTup < totalTuples; startTup += k_ChunkTuples) + { + if(m_ShouldCancel) + { + return {}; + } + const usize count = std::min(k_ChunkTuples, totalTuples - startTup); + eulerStore.copyIntoBuffer(startTup * 3, nonstd::span(buf.data(), count * 3)); - MessageHelper messageHelper(m_MessageHandler); - ProgressMessageHelper progressMessageHelper = messageHelper.createProgressMessageHelper(); - progressMessageHelper.setMaxProgresss(totalElements); - progressMessageHelper.setProgressMessageTemplate("RotateEulerRefFrame: {:.2f}% complete"); + for(usize i = 0; i < count; i++) + { + ebsdlib::OrientationMatrixDType om = ebsdlib::EulerDType(buf[i * 3], buf[i * 3 + 1], buf[i * 3 + 2]).toOrientationMatrix(); + OrientationUtilities::Matrix3dR gNew = (om * rotMat).colwise().normalized(); + ebsdlib::EulerDType eu = ebsdlib::OrientationMatrixDType(gNew.data()).toEuler(); + buf[i * 3] = eu[0]; + buf[i * 3 + 1] = eu[1]; + buf[i * 3 + 2] = eu[2]; + } + + eulerStore.copyFromBuffer(startTup * 3, nonstd::span(buf.data(), count * 3)); + } - // Allow data-based parallelization - ParallelDataAlgorithm dataAlg; - dataAlg.setRange(0, totalElements); - dataAlg.execute(RotateEulerRefFrameImpl(eulerAngles, axis, m_InputValues->rotationAxis[3], m_ShouldCancel, progressMessageHelper)); return {}; } +// ----------------------------------------------------------------------------- bool RotateEulerRefFrame::shouldCancel() const { return m_ShouldCancel; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp index 75761af849..0667a44e71 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp @@ -67,8 +67,8 @@ const std::atomic_bool& WriteGBCDGMTFile::getCancel() // ----------------------------------------------------------------------------- Result<> WriteGBCDGMTFile::operator()() { - auto gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); - auto crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); + auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); // Make sure any directory path is also available as the user may have just typed // in a path without actually creating the full path @@ -143,8 +143,19 @@ Result<> WriteGBCDGMTFile::operator()() // take inverse of misorientation variable to use for switching symmetry Matrix3X3Type dgt = dg.transpose(); + // Cache crystal structures locally (ensemble-level, tiny) + const usize numCrystalStructures = crystalStructures.getSize(); + auto crystalStructuresCache = std::make_unique(numCrystalStructures); + crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.get(), numCrystalStructures)); + + // Cache only the phase-of-interest slice of the GBCD via bulk I/O + const auto totalGBCDBins = (gbcdSizes[0] * gbcdSizes[1] * gbcdSizes[2] * gbcdSizes[3] * gbcdSizes[4] * 2); + const usize phaseOffset = static_cast(m_InputValues->PhaseOfInterest) * static_cast(totalGBCDBins); + auto gbcdPhaseCache = std::make_unique(static_cast(totalGBCDBins)); + gbcd.getDataStoreRef().copyIntoBuffer(phaseOffset, nonstd::span(gbcdPhaseCache.get(), static_cast(totalGBCDBins))); + // Get our LaueOps pointer for the selected crystal structure - const ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructures[m_InputValues->PhaseOfInterest]]; + const ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructuresCache[m_InputValues->PhaseOfInterest]]; // get number of symmetry operators const int32 nSym = orientOps->getNumSymOps(); @@ -160,13 +171,16 @@ Result<> WriteGBCDGMTFile::operator()() const int32 shift3 = gbcdSizes[0] * gbcdSizes[1] * gbcdSizes[2]; const int32 shift4 = gbcdSizes[0] * gbcdSizes[1] * gbcdSizes[2] * gbcdSizes[3]; - const auto totalGBCDBins = (gbcdSizes[0] * gbcdSizes[1] * gbcdSizes[2] * gbcdSizes[3] * gbcdSizes[4] * 2); - std::vector gmtValues; gmtValues.reserve((phiPoints + 1) * (thetaPoints + 1)); // Allocate what should be needed. for(int32 phiPtIndex = 0; phiPtIndex < phiPoints + 1; phiPtIndex++) { + if(m_ShouldCancel) + { + return {}; + } + for(int32 thetaPtIndex = 0; thetaPtIndex < thetaPoints + 1; thetaPtIndex++) { // get (x,y) for stereographic projection pixel @@ -237,7 +251,7 @@ Result<> WriteGBCDGMTFile::operator()() { hemisphere = 1; } - sum += gbcd[(m_InputValues->PhaseOfInterest * totalGBCDBins) + 2 * ((location5 * shift4) + (location4 * shift3) + (location3 * shift2) + (location2 * shift1) + location1) + hemisphere]; + sum += gbcdPhaseCache[2 * ((location5 * shift4) + (location4 * shift3) + (location3 * shift2) + (location2 * shift1) + location1) + hemisphere]; count++; } } @@ -283,7 +297,7 @@ Result<> WriteGBCDGMTFile::operator()() { hemisphere = 1; } - sum += gbcd[(m_InputValues->PhaseOfInterest * totalGBCDBins) + 2 * ((location5 * shift4) + (location4 * shift3) + (location3 * shift2) + (location2 * shift1) + location1) + hemisphere]; + sum += gbcdPhaseCache[2 * ((location5 * shift4) + (location4 * shift3) + (location3 * shift2) + (location2 * shift1) + location1) + hemisphere]; count++; } } @@ -304,9 +318,9 @@ Result<> WriteGBCDGMTFile::operator()() // Remember to use the original Angle in Degrees!!!! fprintf(gmtFilePtr, "%.1f %.1f %.1f %.1f\n", m_InputValues->MisorientationRotation[1], m_InputValues->MisorientationRotation[2], m_InputValues->MisorientationRotation[3], m_InputValues->MisorientationRotation[0]); - const size_t size = gmtValues.size() / 3; + const usize size = gmtValues.size() / 3; - for(size_t i = 0; i < size; i++) + for(usize i = 0; i < size; i++) { fprintf(gmtFilePtr, "%f %f %f\n", gmtValues[3 * i], gmtValues[3 * i + 1], gmtValues[3 * i + 2]); } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp index 3640b156eb..65a9683e32 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp @@ -33,52 +33,61 @@ Result<> WriteGBCDTriangleData::operator()() auto& eulerAngles = m_DataStructure.getDataRefAs(m_InputValues->FeatureEulerAnglesArrayPath); usize numTriangles = faceAreas.getNumberOfTuples(); - FILE* f = fopen(m_InputValues->OutputFile.string().c_str(), "wb"); - if(nullptr == f) + // Cache eulerAngles locally — feature-level (indexed by grain ID, typically thousands) + const usize numEulerElements = eulerAngles.getSize(); + std::vector eulerCache(numEulerElements); + eulerAngles.getDataStoreRef().copyIntoBuffer(0, nonstd::span(eulerCache.data(), numEulerElements)); + + std::ofstream outStream(m_InputValues->OutputFile, std::ios_base::out | std::ios_base::binary); + if(!outStream.is_open()) { return MakeErrorResult(-87000, fmt::format("Error opening output file '{}'", m_InputValues->OutputFile.string())); } - // fprintf(f, "# Triangles Produced from DREAM3D version %s\n", ImportExport::Version::Package().toLatin1().data()); - fprintf(f, "# Column 1-3: right hand average orientation (phi1, PHI, phi2 in RADIANS)\n"); - fprintf(f, "# Column 4-6: left hand average orientation (phi1, PHI, phi2 in RADIANS)\n"); - fprintf(f, "# Column 7-9: triangle normal\n"); - fprintf(f, "# Column 8: surface area\n"); + outStream << "# Column 1-3: right hand average orientation (phi1, PHI, phi2 in RADIANS)\n" + << "# Column 4-6: left hand average orientation (phi1, PHI, phi2 in RADIANS)\n" + << "# Column 7-9: triangle normal\n" + << "# Column 8: surface area\n"; - int32 gid0 = 0; // Feature identifier 0 - int32 gid1 = 0; // Feature identifier 1 - for(int64 t = 0; t < numTriangles; ++t) - { - // Get the Feature Ids for the triangle - gid0 = faceLabels[t * 2]; - gid1 = faceLabels[t * 2 + 1]; + // Process triangles in chunks: bulk-read arrays, format into a string buffer, write once per chunk + constexpr usize k_ChunkSize = 8192; + const auto& labelsStore = faceLabels.getDataStoreRef(); + const auto& normalsStore = faceNormals.getDataStoreRef(); + const auto& areasStore = faceAreas.getDataStoreRef(); + + std::vector labelsBuf(k_ChunkSize * 2); + std::vector normalsBuf(k_ChunkSize * 3); + std::vector areasBuf(k_ChunkSize); + fmt::memory_buffer writeBuf; - if(gid0 < 0) + for(usize chunkStart = 0; chunkStart < numTriangles; chunkStart += k_ChunkSize) + { + if(m_ShouldCancel) { - continue; + return {}; } - if(gid1 < 0) + usize count = std::min(k_ChunkSize, numTriangles - chunkStart); + + labelsStore.copyIntoBuffer(chunkStart * 2, nonstd::span(labelsBuf.data(), count * 2)); + normalsStore.copyIntoBuffer(chunkStart * 3, nonstd::span(normalsBuf.data(), count * 3)); + areasStore.copyIntoBuffer(chunkStart, nonstd::span(areasBuf.data(), count)); + + writeBuf.clear(); + for(usize i = 0; i < count; i++) { - continue; - } + int32 gid0 = labelsBuf[i * 2]; + int32 gid1 = labelsBuf[i * 2 + 1]; - // Now get the Euler Angles for that feature identifier - float32 euAngRightHand0 = eulerAngles[gid0 * 3]; - float32 euAngRightHand1 = eulerAngles[gid0 * 3 + 1]; - float32 euAngRightHand2 = eulerAngles[gid0 * 3 + 2]; - float32 euAngLeftHand0 = eulerAngles[gid1 * 3]; - float32 euAngLeftHand1 = eulerAngles[gid1 * 3 + 1]; - float32 euAngLeftHand2 = eulerAngles[gid1 * 3 + 2]; - - // Get the Triangle Normal - float64 tNorm0 = faceNormals[t * 3]; - float64 tNorm1 = faceNormals[t * 3 + 1]; - float64 tNorm2 = faceNormals[t * 3 + 2]; - - fprintf(f, "%0.4f %0.4f %0.4f %0.4f %0.4f %0.4f %0.4f %0.4f %0.4f %0.4f\n", euAngRightHand0, euAngRightHand1, euAngRightHand2, euAngLeftHand0, euAngLeftHand1, euAngLeftHand2, tNorm0, tNorm1, - tNorm2, faceAreas.getValue(t)); + if(gid0 < 0 || gid1 < 0) + { + continue; + } + + fmt::format_to(std::back_inserter(writeBuf), "{:0.4f} {:0.4f} {:0.4f} {:0.4f} {:0.4f} {:0.4f} {:0.4f} {:0.4f} {:0.4f} {:0.4f}\n", eulerCache[gid0 * 3], eulerCache[gid0 * 3 + 1], + eulerCache[gid0 * 3 + 2], eulerCache[gid1 * 3], eulerCache[gid1 * 3 + 1], eulerCache[gid1 * 3 + 2], normalsBuf[i * 3], normalsBuf[i * 3 + 1], normalsBuf[i * 3 + 2], areasBuf[i]); + } + outStream.write(writeBuf.data(), static_cast(writeBuf.size())); } - fclose(f); return {}; } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.cpp index 7db1fe0e44..7a0796922d 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.cpp @@ -16,7 +16,6 @@ #include "simplnx/Pipeline/Pipeline.hpp" #include "simplnx/Utilities/ArrayCreationUtilities.hpp" #include "simplnx/Utilities/IntersectionUtilities.hpp" -#include "simplnx/Utilities/MaskCompareUtilities.hpp" #include "simplnx/Utilities/ParallelTaskAlgorithm.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" #include "simplnx/Utilities/RTree.hpp" @@ -70,9 +69,9 @@ class ComputeIntensityStereographicProjection { int halfDim = m_Config->imageDim / 2; double* intensity = m_Intensity->getPointer(0); - size_t numCoords = m_XYZCoords->getNumberOfTuples(); - float* xyzPtr = m_XYZCoords->getPointer(0); - for(size_t i = 0; i < numCoords; i++) + usize numCoords = m_XYZCoords->getNumberOfTuples(); + float32* xyzPtr = m_XYZCoords->getPointer(0); + for(usize i = 0; i < numCoords; i++) { if(xyzPtr[i * 3 + 2] < 0.0f) // If the unit sphere data is in the southern hemisphere { @@ -80,13 +79,13 @@ class ComputeIntensityStereographicProjection xyzPtr[i * 3 + 1] *= -1.0f; xyzPtr[i * 3 + 2] *= -1.0f; } - float x = xyzPtr[i * 3] / (1 + xyzPtr[i * 3 + 2]); - float y = xyzPtr[i * 3 + 1] / (1 + xyzPtr[i * 3 + 2]); + float32 x = xyzPtr[i * 3] / (1 + xyzPtr[i * 3 + 2]); + float32 y = xyzPtr[i * 3 + 1] / (1 + xyzPtr[i * 3 + 2]); int xCoord = static_cast(x * static_cast(halfDim - 1)) + halfDim; int yCoord = static_cast(y * static_cast(halfDim - 1)) + halfDim; - size_t index = (yCoord * m_Config->imageDim) + xCoord; + usize index = (yCoord * m_Config->imageDim) + xCoord; intensity[index]++; } @@ -111,25 +110,25 @@ class ComputeIntensityStereographicProjection void unstructuredGridInterpolator(nx::core::IFilter* filter, nx::core::TriangleGeom* delaunayGeom, std::vector& xPositionsPtr, std::vector& yPositionsPtr, T* xyValues, typename std::vector& outputValues) const { - using Vec3f = nx::core::Vec3; - using RTreeType = RTree; + using Vec3f = nx::core::Vec3; + using RTreeType = RTree; // filter->notifyStatusMessage(QString("Starting Interpolation....")); nx::core::IGeometry::SharedFaceList& delTriangles = delaunayGeom->getFacesRef(); - size_t numTriangles = delaunayGeom->getNumberOfFaces(); + usize numTriangles = delaunayGeom->getNumberOfFaces(); // int percent = 0; int counter = xPositionsPtr.size() / 100; RTreeType m_RTree; // Populate the RTree - size_t numTris = delaunayGeom->getNumberOfFaces(); - for(size_t tIndex = 0; tIndex < numTris; tIndex++) + usize numTris = delaunayGeom->getNumberOfFaces(); + for(usize tIndex = 0; tIndex < numTris; tIndex++) { - std::array boundBox = nx::core::IntersectionUtilities::GetBoundingBoxAtTri(*delaunayGeom, tIndex); + std::array boundBox = nx::core::IntersectionUtilities::GetBoundingBoxAtTri(*delaunayGeom, tIndex); m_RTree.Insert(boundBox.data(), boundBox.data() + 3, tIndex); // Note, all values including zero are fine in this version } - for(size_t vertIndex = 0; vertIndex < xPositionsPtr.size(); vertIndex++) + for(usize vertIndex = 0; vertIndex < xPositionsPtr.size(); vertIndex++) { Vec3f rayOrigin(xPositionsPtr[vertIndex], yPositionsPtr[vertIndex], 1.0F); Vec3f rayDirection(0.0F, 0.0F, -1.0F); @@ -149,8 +148,8 @@ class ComputeIntensityStereographicProjection // Create these reusable variables to save the reallocation each time through the loop - std::vector hitTriangleIds; - std::function func = [&](size_t id) { + std::vector hitTriangleIds; + std::function func = [&](usize id) { hitTriangleIds.push_back(id); return true; // keep going }; @@ -159,7 +158,7 @@ class ComputeIntensityStereographicProjection for(auto triIndex : hitTriangleIds) { barycentricCoord = {0.0F, 0.0F, 0.0F}; - std::array triVertIndices; + std::array triVertIndices; // Get the Vertex Coordinates for each of the 3 vertices std::array verts; delaunayGeom->getFaceCoordinates(triIndex, verts); @@ -174,11 +173,11 @@ class ComputeIntensityStereographicProjection { // Linear Interpolate dx and dy values using the barycentric coordinates delaunayGeom->getFaceCoordinates(triIndex, verts); - float f0 = xyValues[triVertIndices[0]]; - float f1 = xyValues[triVertIndices[1]]; - float f2 = xyValues[triVertIndices[2]]; + float32 f0 = xyValues[triVertIndices[0]]; + float32 f1 = xyValues[triVertIndices[1]]; + float32 f2 = xyValues[triVertIndices[2]]; - float interpolatedVal = (barycentricCoord[0] * f0) + (barycentricCoord[1] * f1) + (barycentricCoord[2] * f2); + float32 interpolatedVal = (barycentricCoord[0] * f0) + (barycentricCoord[1] * f1) + (barycentricCoord[2] * f2); outputValues[vertIndex] = interpolatedVal; @@ -198,34 +197,34 @@ class ComputeIntensityStereographicProjection // We want half the sphere area for each square because each square represents a hemisphere. const float32 sphereRadius = 1.0f; - float halfSphereArea = 4.0f * ebsdlib::constants::k_PiF * sphereRadius * sphereRadius / 2.0f; + float32 halfSphereArea = 4.0f * ebsdlib::constants::k_PiF * sphereRadius * sphereRadius / 2.0f; // The length of a side of the square is the square root of the area - float squareEdge = std::sqrt(halfSphereArea); - float32 m_StepSize = squareEdge / static_cast(m_Dimension); + float32 squareEdge = std::sqrt(halfSphereArea); + float32 m_StepSize = squareEdge / static_cast(m_Dimension); float32 m_MaxCoord = squareEdge / 2.0f; float32 m_MinCoord = -squareEdge / 2.0f; - std::array vert = {0.0f, 0.0f, 0.0f}; + std::array vert = {0.0f, 0.0f, 0.0f}; - std::vector squareCoords(m_Dimension * m_Dimension * 3); + std::vector squareCoords(m_Dimension * m_Dimension * 3); // Northern Hemisphere Coordinates - std::vector northSphereCoords(m_Dimension * m_Dimension * 3); - std::vector northStereoCoords(m_Dimension * m_Dimension * 3); + std::vector northSphereCoords(m_Dimension * m_Dimension * 3); + std::vector northStereoCoords(m_Dimension * m_Dimension * 3); // Southern Hemisphere Coordinates - std::vector southSphereCoords(m_Dimension * m_Dimension * 3); - std::vector southStereoCoords(m_Dimension * m_Dimension * 3); + std::vector southSphereCoords(m_Dimension * m_Dimension * 3); + std::vector southStereoCoords(m_Dimension * m_Dimension * 3); - size_t index = 0; + usize index = 0; - const float origin = m_MinCoord + (m_StepSize / 2.0f); - for(int32_t y = 0; y < m_Dimension; ++y) + const float32 origin = m_MinCoord + (m_StepSize / 2.0f); + for(int32 y = 0; y < m_Dimension; ++y) { for(int x = 0; x < m_Dimension; ++x) { - vert[0] = origin + (static_cast(x) * m_StepSize); - vert[1] = origin + (static_cast(y) * m_StepSize); + vert[0] = origin + (static_cast(x) * m_StepSize); + vert[1] = origin + (static_cast(y) * m_StepSize); squareCoords[index * 3] = vert[0]; squareCoords[index * 3 + 1] = vert[1]; @@ -242,8 +241,8 @@ class ComputeIntensityStereographicProjection northStereoCoords[index * 3 + 2] = 0.0f; // Reset the Lambert Square Coord - vert[0] = origin + (static_cast(x) * m_StepSize); - vert[1] = origin + (static_cast(y) * m_StepSize); + vert[0] = origin + (static_cast(x) * m_StepSize); + vert[1] = origin + (static_cast(y) * m_StepSize); ebsdlib::LambertUtilities::LambertSquareVertToSphereVert(vert.data(), ebsdlib::LambertUtilities::Hemisphere::South); southSphereCoords[index * 3] = vert[0]; @@ -271,7 +270,7 @@ class ComputeIntensityStereographicProjection usize numPts = northStereoCoords.size() / 3; // Create the default DataArray that will hold the FaceList and Vertices. We // size these to 1 because the Csv parser will resize them to the appropriate number of tuples - using DimensionType = std::vector; + using DimensionType = ShapeType; DimensionType faceTupleShape = {0}; Result result = ArrayCreationUtilities::CreateArray(dataStructure, faceTupleShape, {3ULL}, sharedFaceListPath, IDataAction::Mode::Execute); @@ -287,7 +286,7 @@ class ComputeIntensityStereographicProjection DataPath vertexPath({"Delaunay", "SharedVertexList"}); DimensionType vertexTupleShape = {0}; - result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, IDataAction::Mode::Execute); + result = ArrayCreationUtilities::CreateArray(dataStructure, vertexTupleShape, {3}, vertexPath, IDataAction::Mode::Execute); if(result.invalid()) { return -2; @@ -357,15 +356,15 @@ class ComputeIntensityStereographicProjection //****************************************************************************************************************************** // Perform a Bi-linear Interpolation // Generate a regular grid of XY points - size_t numSteps = 1024; - float32 stepInc = 2.0f / static_cast(numSteps); + usize numSteps = 1024; + float32 stepInc = 2.0f / static_cast(numSteps); std::vector xcoords(numSteps * numSteps); std::vector ycoords(numSteps * numSteps); - for(size_t y = 0; y < numSteps; ++y) + for(usize y = 0; y < numSteps; ++y) { - for(size_t x = 0; x < numSteps; x++) + for(usize x = 0; x < numSteps; x++) { - size_t idx = y * numSteps + x; + usize idx = y * numSteps + x; xcoords[idx] = -1.0f + static_cast(x) * stepInc; ycoords[idx] = -1.0f + static_cast(y) * stepInc; } @@ -439,13 +438,13 @@ std::vector createIntensityPoleFigures(ebsdli label2 = config.labels.at(2); } - const size_t numOrientations = config.eulers->getNumberOfTuples(); + const usize numOrientations = config.eulers->getNumberOfTuples(); // Create an Array to hold the XYZ Coordinates which are the coords on the sphere. // this is size for CUBIC ONLY, <001> Family - std::array symSize = ops.getNumSymmetry(); + std::array symSize = ops.getNumSymmetry(); - const std::vector dims = {3}; + const ShapeType dims = {3}; const ebsdlib::FloatArrayType::Pointer xyz001 = ebsdlib::FloatArrayType::CreateArray(numOrientations * symSize[0], dims, label0 + std::string("xyzCoords"), true); // this is size for CUBIC ONLY, <011> Family const ebsdlib::FloatArrayType::Pointer xyz011 = ebsdlib::FloatArrayType::CreateArray(numOrientations * symSize[1], dims, label1 + std::string("xyzCoords"), true); @@ -485,8 +484,8 @@ typename EbsdDataArray::Pointer flipAndMirrorPoleFigure(EbsdDataArray* src const int destY = config.imageDim - 1 - y; for(int x = 0; x < config.imageDim; x++) { - const size_t indexSrc = y * config.imageDim + x; - const size_t indexDest = destY * config.imageDim + x; + const usize indexSrc = y * config.imageDim + x; + const usize indexDest = destY * config.imageDim + x; T* argbPtr = src->getTuplePointer(indexSrc); converted->setTuple(indexDest, argbPtr); @@ -501,8 +500,8 @@ typename EbsdDataArray::Pointer convertColorOrder(EbsdDataArray* src, cons typename EbsdDataArray::Pointer converted = EbsdDataArray::CreateArray(config.imageDim * config.imageDim, src->getComponentDimensions(), src->getName(), true); // BGRA to RGBA ordering (This is a Little Endian code) // If this is ever compiled on a BIG ENDIAN machine the colors will be off. - size_t numTuples = src->getNumberOfTuples(); - for(size_t tIdx = 0; tIdx < numTuples; tIdx++) + usize numTuples = src->getNumberOfTuples(); + for(usize tIdx = 0; tIdx < numTuples; tIdx++) { T* argbPtr = src->getTuplePointer(tIdx); T* destPtr = converted->getTuplePointer(tIdx); @@ -515,17 +514,17 @@ typename EbsdDataArray::Pointer convertColorOrder(EbsdDataArray* src, cons } // ----------------------------------------------------------------------------- -void drawInformationBlock(canvas_ity::canvas& context, const ebsdlib::PoleFigureConfiguration_t& config, const std::pair& position, float margins, float fontPtSize, int32_t phaseNum, - std::vector& fontData, const std::string& laueGroupName, const std::string& materialName) +void drawInformationBlock(canvas_ity::canvas& context, const ebsdlib::PoleFigureConfiguration_t& config, const std::pair& position, float32 margins, float32 fontPtSize, + int32 phaseNum, std::vector& fontData, const std::string& laueGroupName, const std::string& materialName) { - const float scaleBarRelativeWidth = 0.10f; + const float32 scaleBarRelativeWidth = 0.10f; // const int imageHeight = config.imageDim; const int imageWidth = config.imageDim; - const float colorHeight = (static_cast(imageHeight)) / static_cast(config.numColors); + const float32 colorHeight = (static_cast(imageHeight)) / static_cast(config.numColors); // - using RectFType = std::pair; - const RectFType rect = std::make_pair(static_cast(imageWidth) * scaleBarRelativeWidth, colorHeight * 1.00000f); + using RectFType = std::pair; + const RectFType rect = std::make_pair(static_cast(imageWidth) * scaleBarRelativeWidth, colorHeight * 1.00000f); // const std::array baselines = {canvas_ity::alphabetic, canvas_ity::top, canvas_ity::middle, canvas_ity::bottom, canvas_ity::hanging, canvas_ity::ideographic}; @@ -541,7 +540,7 @@ void drawInformationBlock(canvas_ity::canvas& context, const ebsdlib::PoleFigure }; /* clang-format on */ - float heightInc = 1.0f; + float32 heightInc = 1.0f; for(const auto& label : labels) { // Draw the Number of Samples @@ -549,14 +548,14 @@ void drawInformationBlock(canvas_ity::canvas& context, const ebsdlib::PoleFigure context.set_font(fontData.data(), static_cast(fontData.size()), fontPtSize); context.set_color(canvas_ity::fill_style, 0.0f, 0.0f, 0.0f, 1.0f); context.text_baseline = baselines[0]; - context.fill_text(label.c_str(), position.first + margins + rect.first + margins, position.second + margins + (static_cast(imageHeight) / 3.0f) + (heightInc * fontPtSize)); + context.fill_text(label.c_str(), position.first + margins + rect.first + margins, position.second + margins + (static_cast(imageHeight) / 3.0f) + (heightInc * fontPtSize)); context.close_path(); heightInc++; } } // ----------------------------------------------------------------------------- -void drawScalarBar(canvas_ity::canvas& context, const ebsdlib::PoleFigureConfiguration_t& config, const std::pair& position, float margins, float fontPtSize, int32_t phaseNum, +void drawScalarBar(canvas_ity::canvas& context, const ebsdlib::PoleFigureConfiguration_t& config, const std::pair& position, float32 margins, float32 fontPtSize, int32 phaseNum, std::vector& fontData, const std::string& laueGroupName, const std::string& materialName) { @@ -564,11 +563,11 @@ void drawScalarBar(canvas_ity::canvas& context, const ebsdlib::PoleFigureConfigu // Get all the colors that we will need std::vector colorTable(numColors); - std::vector colors(3 * numColors, 0.0); + std::vector colors(3 * numColors, 0.0); nx::core::RgbColor::GetColorTable(numColors, colors); // Generate the color table values - float r = 0.0; - float g = 0.0; - float b = 0.0; + float32 r = 0.0; + float32 g = 0.0; + float32 b = 0.0; for(int i = 0; i < numColors; i++) // Convert them to QRgbColor values { r = colors[3 * i]; @@ -579,13 +578,13 @@ void drawScalarBar(canvas_ity::canvas& context, const ebsdlib::PoleFigureConfigu // Now start from the bottom and draw colored lines up the scale bar // A Slight Indentation for the scalar bar - const float scaleBarRelativeWidth = 0.10f; + const float32 scaleBarRelativeWidth = 0.10f; const int imageHeight = config.imageDim; const int imageWidth = config.imageDim; - const float colorHeight = (static_cast(imageHeight)) / static_cast(numColors); + const float32 colorHeight = (static_cast(imageHeight)) / static_cast(numColors); - using RectFType = std::pair; + using RectFType = std::pair; const RectFType rect = std::make_pair(static_cast(imageWidth) * scaleBarRelativeWidth, colorHeight * 1.00000f); @@ -680,25 +679,63 @@ Result<> WritePoleFigure::operator()() auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); auto& materialNames = m_DataStructure.getDataRefAs(m_InputValues->MaterialNameArrayPath); - std::unique_ptr maskCompare = nullptr; - if(m_InputValues->UseMask) + // Find the total number of angles we have based on the number of Tuples of the + // Euler Angles array + const usize numPoints = eulerAngles.getNumberOfTuples(); + // Find how many phases we have by getting the number of Crystal Structures + const usize numPhases = crystalStructures.getNumberOfTuples(); + + // Get DataStore references for chunked iteration (no O(n) pre-caching) + const auto& eulerStoreRef = eulerAngles.getDataStoreRef(); + const auto& phasesStoreRef = phases.getDataStoreRef(); + + const bool useMask = m_InputValues->UseMask; + bool maskIsBool = false; + const AbstractDataStore* boolMaskStorePtr = nullptr; + const AbstractDataStore* uint8MaskStorePtr = nullptr; + if(useMask) { - try + auto* maskArray = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); + if(maskArray != nullptr) { - maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); - } catch(const std::out_of_range& exception) - { - // This really should NOT be happening as the path was verified during preflight BUT we may be calling this from - // some other context that is NOT going through the normal nx::core::IFilter API of Preflight and Execute - return MakeErrorResult(-53900, fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString())); + maskIsBool = (maskArray->getDataType() == DataType::boolean); + if(maskIsBool) + { + boolMaskStorePtr = &m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath).getDataStoreRef(); + } + else + { + uint8MaskStorePtr = &m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath).getDataStoreRef(); + } } } - // Find the total number of angles we have based on the number of Tuples of the - // Euler Angles array - const size_t numPoints = eulerAngles.getNumberOfTuples(); - // Find how many phases we have by getting the number of Crystal Structures - const size_t numPhases = crystalStructures.getNumberOfTuples(); + // Bounded chunk buffers for iterating cell-level arrays (reused across phases) + constexpr usize k_ChunkTuples = 65536; + auto eulerChunk = std::make_unique(k_ChunkTuples * 3); + auto phasesChunk = std::make_unique(k_ChunkTuples); + auto maskChunk = std::make_unique(useMask ? k_ChunkTuples : 1); + auto boolScratch = std::make_unique(useMask && maskIsBool ? k_ChunkTuples : 1); + + // Helper lambda: read mask chunk and convert bool->uint8 if needed + auto readMaskChunk = [&](usize offset, usize chunkSize) { + if(!useMask) + { + return; + } + if(maskIsBool) + { + boolMaskStorePtr->copyIntoBuffer(offset, nonstd::span(boolScratch.get(), chunkSize)); + for(usize i = 0; i < chunkSize; i++) + { + maskChunk[i] = boolScratch[i] ? 1 : 0; + } + } + else + { + uint8MaskStorePtr->copyIntoBuffer(offset, nonstd::span(maskChunk.get(), chunkSize)); + } + }; // Create the Image Geometry that will serve as the final storage location for each // pole figure. We are just giving it a default size for now, it will be resized @@ -710,37 +747,49 @@ Result<> WritePoleFigure::operator()() imageGeom.getCellData()->resizeTuples(tupleShape); // Loop over all the voxels gathering the Euler angles for a specific phase into an array - for(size_t phase = 1; phase < numPhases; ++phase) + for(usize phase = 1; phase < numPhases; ++phase) { - size_t count = 0; - // First find out how many voxels we are going to have. This is probably faster to loop twice than to - // keep allocating memory everytime we find one. - for(size_t i = 0; i < numPoints; ++i) + if(m_ShouldCancel) + { + return {}; + } + + // Chunked count pass: find how many voxels belong to this phase + usize count = 0; + for(usize offset = 0; offset < numPoints; offset += k_ChunkTuples) { - if(phases[i] == phase) + const usize chunkSize = std::min(k_ChunkTuples, numPoints - offset); + phasesStoreRef.copyIntoBuffer(offset, nonstd::span(phasesChunk.get(), chunkSize)); + readMaskChunk(offset, chunkSize); + for(usize i = 0; i < chunkSize; i++) { - if(!m_InputValues->UseMask || maskCompare->isTrue(i)) + if(phasesChunk[i] == static_cast(phase) && (!useMask || maskChunk[i])) { count++; } } } - const std::vector eulerCompDim = {3}; + + const ShapeType eulerCompDim = {3}; const ebsdlib::FloatArrayType::Pointer subEulerAnglesPtr = ebsdlib::FloatArrayType::CreateArray(count, eulerCompDim, "Euler_Angles_Per_Phase", true); - subEulerAnglesPtr->initializeWithValue(std::numeric_limits::signaling_NaN()); + subEulerAnglesPtr->initializeWithValue(std::numeric_limits::signaling_NaN()); ebsdlib::FloatArrayType& subEulerAngles = *subEulerAnglesPtr; - // Now loop through the Euler angles again and this time add them to the sub-Euler angle Array + // Chunked fill pass: copy matching Euler angles into the sub-array count = 0; - for(size_t i = 0; i < numPoints; ++i) + for(usize offset = 0; offset < numPoints; offset += k_ChunkTuples) { - if(phases[i] == phase) + const usize chunkSize = std::min(k_ChunkTuples, numPoints - offset); + phasesStoreRef.copyIntoBuffer(offset, nonstd::span(phasesChunk.get(), chunkSize)); + eulerStoreRef.copyIntoBuffer(offset * 3, nonstd::span(eulerChunk.get(), chunkSize * 3)); + readMaskChunk(offset, chunkSize); + for(usize i = 0; i < chunkSize; i++) { - if(!m_InputValues->UseMask || maskCompare->isTrue(i)) + if(phasesChunk[i] == static_cast(phase) && (!useMask || maskChunk[i])) { - subEulerAngles[count * 3] = eulerAngles[i * 3]; - subEulerAngles[count * 3 + 1] = eulerAngles[i * 3 + 1]; - subEulerAngles[count * 3 + 2] = eulerAngles[i * 3 + 2]; + subEulerAngles[count * 3] = eulerChunk[i * 3]; + subEulerAngles[count * 3 + 1] = eulerChunk[i * 3 + 1]; + subEulerAngles[count * 3 + 2] = eulerChunk[i * 3 + 2]; count++; } } @@ -823,7 +872,7 @@ Result<> WritePoleFigure::operator()() // If there is more than a single phase we will need to add more arrays to the DataStructure if(phase > 1) { - const std::vector intensityImageDims = {static_cast(config.imageDim), static_cast(config.imageDim), 1ULL}; + const ShapeType intensityImageDims = {static_cast(config.imageDim), static_cast(config.imageDim), 1ULL}; DataPath arrayDataPath = amPath.createChildPath(fmt::format("Phase_{}_{}", phase, m_InputValues->IntensityPlot1Name)); Result<> result = ArrayCreationUtilities::CreateArray(m_DataStructure, intensityImageDims, {1ULL}, arrayDataPath, IDataAction::Mode::Execute); @@ -838,25 +887,28 @@ Result<> WritePoleFigure::operator()() auto intensityPlot2Array = m_DataStructure.getDataRefAs(amPath.createChildPath(fmt::format("Phase_{}_{}", phase, m_InputValues->IntensityPlot2Name))); auto intensityPlot3Array = m_DataStructure.getDataRefAs(amPath.createChildPath(fmt::format("Phase_{}_{}", phase, m_InputValues->IntensityPlot3Name))); - std::vector compDims = {1ULL}; + ShapeType compDims = {1ULL}; for(int imageIndex = 0; imageIndex < figures.size(); imageIndex++) { intensityImages[imageIndex] = flipAndMirrorPoleFigure(intensityImages[imageIndex].get(), config); } - std::copy(intensityImages[0]->begin(), intensityImages[0]->end(), intensityPlot1Array.begin()); - std::copy(intensityImages[1]->begin(), intensityImages[1]->end(), intensityPlot2Array.begin()); - std::copy(intensityImages[2]->begin(), intensityImages[2]->end(), intensityPlot3Array.begin()); + { + const usize numElements = intensityPlot1Array.getSize(); + intensityPlot1Array.getDataStoreRef().copyFromBuffer(0, nonstd::span(intensityImages[0]->getPointer(0), numElements)); + intensityPlot2Array.getDataStoreRef().copyFromBuffer(0, nonstd::span(intensityImages[1]->getPointer(0), numElements)); + intensityPlot3Array.getDataStoreRef().copyFromBuffer(0, nonstd::span(intensityImages[2]->getPointer(0), numElements)); + } DataPath metaDataPath = m_InputValues->IntensityGeometryDataPath.createChildPath(write_pole_figure::k_MetaDataName); auto metaDataArrayRef = m_DataStructure.getDataRefAs(metaDataPath); if(metaDataArrayRef.getNumberOfTuples() != numPhases) { - metaDataArrayRef.resizeTuples(std::vector{numPhases}); + metaDataArrayRef.resizeTuples(ShapeType{numPhases}); } std::vector laueNames = ebsdlib::LaueOps::GetLaueNames(); - const uint32_t laueIndex = crystalStructures[phase]; + const uint32 laueIndex = crystalStructures[phase]; const std::string materialName = materialNames[phase]; metaDataArrayRef[phase] = fmt::format("Phase Num: {}\nMaterial Name: {}\nLaue Group: {}\nHemisphere: Northern\nSamples: {}\nLambert Square Dim: {}", phase, materialName, laueNames[laueIndex], @@ -867,8 +919,8 @@ Result<> WritePoleFigure::operator()() { const auto imageWidth = static_cast(config.imageDim); const auto imageHeight = static_cast(config.imageDim); - const float32 fontPtSize = static_cast(imageHeight) / 16.0f; - const float32 margins = static_cast(imageHeight) / 32.0f; + const float32 fontPtSize = static_cast(imageHeight) / 16.0f; + const float32 margins = static_cast(imageHeight) / 32.0f; int32 pageWidth = 0; auto pageHeight = static_cast(margins + fontPtSize); @@ -889,10 +941,10 @@ Result<> WritePoleFigure::operator()() { pageWidth = static_cast(subCanvasWidth) * 4; pageHeight = pageHeight + static_cast(subCanvasHeight); - globalImageOrigins[0] = std::make_pair(0.0f, static_cast(pageHeight) - subCanvasHeight); - globalImageOrigins[1] = std::make_pair(subCanvasWidth, static_cast(pageHeight) - subCanvasHeight); - globalImageOrigins[2] = std::make_pair(subCanvasWidth * 2.0f, static_cast(pageHeight) - subCanvasHeight); - globalImageOrigins[3] = std::make_pair(subCanvasWidth * 3.0f, static_cast(pageHeight) - subCanvasHeight); + globalImageOrigins[0] = std::make_pair(0.0f, static_cast(pageHeight) - subCanvasHeight); + globalImageOrigins[1] = std::make_pair(subCanvasWidth, static_cast(pageHeight) - subCanvasHeight); + globalImageOrigins[2] = std::make_pair(subCanvasWidth * 2.0f, static_cast(pageHeight) - subCanvasHeight); + globalImageOrigins[3] = std::make_pair(subCanvasWidth * 3.0f, static_cast(pageHeight) - subCanvasHeight); } else if(static_cast(m_InputValues->ImageLayout) == WritePoleFigure::LayoutType::Vertical) { @@ -907,10 +959,10 @@ Result<> WritePoleFigure::operator()() { pageWidth = static_cast(subCanvasWidth) * 2; pageHeight = pageHeight + static_cast(subCanvasHeight) * 2; - globalImageOrigins[0] = std::make_pair(0.0f, (static_cast(pageHeight) - 2.0f * subCanvasHeight)); // Upper Left - globalImageOrigins[1] = std::make_pair(subCanvasWidth, (static_cast(pageHeight) - 2.0f * subCanvasHeight)); // Upper Right - globalImageOrigins[2] = std::make_pair(0.0f, (static_cast(pageHeight) - subCanvasHeight)); // Lower Left - globalImageOrigins[3] = std::make_pair(subCanvasWidth, (static_cast(pageHeight) - subCanvasHeight)); // Lower Right + globalImageOrigins[0] = std::make_pair(0.0f, (static_cast(pageHeight) - 2.0f * subCanvasHeight)); // Upper Left + globalImageOrigins[1] = std::make_pair(subCanvasWidth, (static_cast(pageHeight) - 2.0f * subCanvasHeight)); // Upper Right + globalImageOrigins[2] = std::make_pair(0.0f, (static_cast(pageHeight) - subCanvasHeight)); // Lower Left + globalImageOrigins[3] = std::make_pair(subCanvasWidth, (static_cast(pageHeight) - subCanvasHeight)); // Lower Right } // Create a Canvas to draw into @@ -923,15 +975,15 @@ Result<> WritePoleFigure::operator()() // Fill the whole background with white context.move_to(0.0f, 0.0f); - context.line_to(static_cast(pageWidth), 0.0f); - context.line_to(static_cast(pageWidth), static_cast(pageHeight)); - context.line_to(0.0f, static_cast(pageHeight)); + context.line_to(static_cast(pageWidth), 0.0f); + context.line_to(static_cast(pageWidth), static_cast(pageHeight)); + context.line_to(0.0f, static_cast(pageHeight)); context.line_to(0.0f, 0.0f); context.close_path(); context.set_color(canvas_ity::fill_style, 1.0f, 1.0f, 1.0f, 1.0f); context.fill(); - std::vector compDims = {4ULL}; + ShapeType compDims = {4ULL}; for(int imageIndex = 0; imageIndex < figures.size(); imageIndex++) { figures[imageIndex] = flipAndMirrorPoleFigure(figures[imageIndex].get(), config); @@ -940,7 +992,7 @@ Result<> WritePoleFigure::operator()() for(int i = 0; i < 3; i++) { - std::array figureOrigin = {0.0f, 0.0f}; + std::array figureOrigin = {0.0f, 0.0f}; std::tie(figureOrigin[0], figureOrigin[1]) = globalImageOrigins[i]; context.draw_image(figures[i]->getPointer(0), imageWidth, imageHeight, imageWidth * figures[i]->getNumberOfComponents(), figureOrigin[0] + margins, figureOrigin[1] + fontPtSize * 2.0f + margins * 2.0f, static_cast(imageWidth), static_cast(imageHeight)); @@ -952,7 +1004,7 @@ Result<> WritePoleFigure::operator()() context.set_color(canvas_ity::stroke_style, 0.0f, 0.0f, 0.0f, 1.0f); context.arc(figureOrigin[0] + margins + static_cast(m_InputValues->ImageSize) / 2.0f, figureOrigin[1] + fontPtSize * 2.0f + margins * 2.0f + static_cast(m_InputValues->ImageSize) / 2.0f, static_cast(m_InputValues->ImageSize) / 2.0f, 0, - nx::core::Constants::k_2Pi); + nx::core::Constants::k_2Pi); context.stroke(); context.close_path(); @@ -992,7 +1044,7 @@ Result<> WritePoleFigure::operator()() context.set_font(m_LatoBold.data(), static_cast(m_LatoBold.size()), fontPtSize); context.set_color(canvas_ity::fill_style, 0.0f, 0.0f, 0.0f, 1.0f); context.text_baseline = baselines[0]; - const float yFontWidth = context.measure_text("Y"); + const float32 yFontWidth = context.measure_text("Y"); context.fill_text("Y", figureOrigin[0] + margins - (0.5f * yFontWidth) + static_cast(m_InputValues->ImageSize) / 2.0f, figureOrigin[1] + fontPtSize * 2.0f + margins); context.close_path(); @@ -1001,21 +1053,21 @@ Result<> WritePoleFigure::operator()() figureSubtitle = nx::core::StringUtilities::replace(figureSubtitle, "<", "("); figureSubtitle = nx::core::StringUtilities::replace(figureSubtitle, ">", ")"); std::string bottomPart; - std::array textOrigin = {figureOrigin[0] + margins, figureOrigin[1] + fontPtSize + 2 * margins}; - for(size_t idx = 0; idx < figureSubtitle.size(); idx++) + std::array textOrigin = {figureOrigin[0] + margins, figureOrigin[1] + fontPtSize + 2 * margins}; + for(usize idx = 0; idx < figureSubtitle.size(); idx++) { if(figureSubtitle.at(idx) == '-') { const char charBuf[] = {figureSubtitle[idx + 1], 0}; context.set_font(m_FiraSansRegular.data(), static_cast(m_FiraSansRegular.size()), fontPtSize); - float tw = 0.0f; + float32 tw = 0.0f; if(!bottomPart.empty()) { tw = context.measure_text(bottomPart.c_str()); } - const float charWidth = context.measure_text(charBuf); - const float dashWidth = charWidth * 0.5f; - const float dashOffset = charWidth * 0.25f; + const float32 charWidth = context.measure_text(charBuf); + const float32 dashWidth = charWidth * 0.5f; + const float32 dashOffset = charWidth * 0.25f; context.begin_path(); context.line_cap = canvas_ity::square; @@ -1048,22 +1100,21 @@ Result<> WritePoleFigure::operator()() context.fill_text(m_InputValues->Title.c_str(), margins, margins + fontPtSize); std::vector laueNames = ebsdlib::LaueOps::GetLaueNames(); - const uint32_t laueIndex = crystalStructures[phase]; + const uint32 laueIndex = crystalStructures[phase]; const std::string materialName = materialNames[phase]; // Now draw the Color Scalar Bar if needed. if(config.discrete) { - drawInformationBlock(context, config, globalImageOrigins[3], margins, static_cast(imageHeight) / 20.0f, static_cast(phase), m_LatoRegular, laueNames[laueIndex], - materialName); + drawInformationBlock(context, config, globalImageOrigins[3], margins, static_cast(imageHeight) / 20.0f, static_cast(phase), m_LatoRegular, laueNames[laueIndex], materialName); } else { - drawScalarBar(context, config, globalImageOrigins[3], margins, static_cast(imageHeight) / 20.0f, static_cast(phase), m_LatoRegular, laueNames[laueIndex], materialName); + drawScalarBar(context, config, globalImageOrigins[3], margins, static_cast(imageHeight) / 20.0f, static_cast(phase), m_LatoRegular, laueNames[laueIndex], materialName); } // Fetch the rendered RGBA pixels from the entire canvas. - std::vector rgbaCanvasImage(static_cast(pageHeight * pageWidth * 4)); + std::vector rgbaCanvasImage(static_cast(pageHeight * pageWidth * 4)); context.get_image_data(rgbaCanvasImage.data(), pageWidth, pageHeight, pageWidth * 4, 0, 0); if(m_InputValues->SaveAsImageGeometry) { @@ -1085,14 +1136,15 @@ Result<> WritePoleFigure::operator()() // canvas RGBA data. auto& imageData = m_DataStructure.getDataRefAs(imageArrayPath); - imageData.fill(0); - size_t tupleCount = pageHeight * pageWidth; - for(size_t t = 0; t < tupleCount; t++) + const usize tupleCount = pageHeight * pageWidth; + std::vector rgbBuf(tupleCount * 3); + for(usize t = 0; t < tupleCount; t++) { - imageData[t * 3 + 0] = rgbaCanvasImage[t * 4 + 0]; - imageData[t * 3 + 1] = rgbaCanvasImage[t * 4 + 1]; - imageData[t * 3 + 2] = rgbaCanvasImage[t * 4 + 2]; + rgbBuf[t * 3 + 0] = rgbaCanvasImage[t * 4 + 0]; + rgbBuf[t * 3 + 1] = rgbaCanvasImage[t * 4 + 1]; + rgbBuf[t * 3 + 2] = rgbaCanvasImage[t * 4 + 2]; } + imageData.getDataStoreRef().copyFromBuffer(0, nonstd::span(rgbBuf.data(), rgbBuf.size())); } // Write out the full RGBA data diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp index 3b86e6a7ae..5f817313bb 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/BadDataNeighborOrientationCheckFilter.cpp @@ -1,5 +1,5 @@ #include "BadDataNeighborOrientationCheckFilter.hpp" -#include "OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp" +#include "OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -135,7 +135,7 @@ Result<> BadDataNeighborOrientationCheckFilter::executeImpl(DataStructure& dataS inputValues.CellPhasesArrayPath = filterArgs.value(k_CellPhasesArrayPath_Key); inputValues.CrystalStructuresArrayPath = filterArgs.value(k_CrystalStructuresArrayPath_Key); - return BadDataNeighborOrientationCheckWorklist(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return BadDataNeighborOrientationCheck(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeGBCDPoleFigureFilter.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeGBCDPoleFigureFilter.cpp index 84c42dbc6a..c5e2a4d0c1 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeGBCDPoleFigureFilter.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/ComputeGBCDPoleFigureFilter.cpp @@ -1,5 +1,8 @@ #include "ComputeGBCDPoleFigureFilter.hpp" -#include "OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigure.hpp" +#include "OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp" +#include "OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp" + +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataPath.hpp" @@ -143,7 +146,8 @@ Result<> ComputeGBCDPoleFigureFilter::executeImpl(DataStructure& dataStructure, inputValues.CellAttributeMatrixName = filterArgs.value(k_CellAttributeMatrixName_Key); inputValues.CellIntensityArrayName = filterArgs.value(k_CellIntensityArrayName_Key); - return ComputeGBCDPoleFigure(dataStructure, messageHandler, shouldCancel, &inputValues)(); + auto* gbcdArray = dataStructure.getDataAs(inputValues.GBCDArrayPath); + return DispatchAlgorithm({gbcdArray}, dataStructure, messageHandler, shouldCancel, &inputValues); } namespace diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/NeighborOrientationCorrelationFilter.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/NeighborOrientationCorrelationFilter.hpp index 22a64b688f..46e0e13e75 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/NeighborOrientationCorrelationFilter.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/NeighborOrientationCorrelationFilter.hpp @@ -9,7 +9,21 @@ namespace nx::core { /** * @class NeighborOrientationCorrelationFilter - * @brief This filter will .... + * @brief Cleans up EBSD data by replacing low-confidence voxels with data from + * the most orientation-correlated face neighbor. + * + * This filter identifies voxels whose confidence index falls below a + * user-specified threshold, then examines the 6 face neighbors of each such + * voxel. Neighbor pairs are compared using crystallographic misorientation; + * the neighbor that agrees most with the other neighbors (within a given + * angular tolerance) is selected as the replacement source. All cell-level + * DataArrays are updated to reflect the replacement. The process repeats + * across multiple cleanup levels, progressively relaxing the required neighbor + * agreement count. + * + * The underlying algorithm uses Z-slice buffering to maintain efficient + * sequential access patterns, which is critical for out-of-core (OOC) data + * where arrays are stored as compressed Zarr chunks on disk. */ class ORIENTATIONANALYSIS_EXPORT NeighborOrientationCorrelationFilter : public IFilter { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp index ee23773c50..422cc8361f 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/IEbsdOemReader.hpp @@ -57,10 +57,16 @@ class ORIENTATIONANALYSIS_EXPORT IEbsdOemReader auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); imageGeom.setUnits(IGeometry::LengthUnit::Micrometer); + const auto& scanNames = m_InputValues->SelectedScanNames.scanNames; int index = 0; - for(const auto& currentScanName : m_InputValues->SelectedScanNames.scanNames) + for(const auto& currentScanName : scanNames) { - m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Importing Index {}", currentScanName)}); + if(m_ShouldCancel) + { + return {}; + } + + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Importing scan {}/{}: '{}'", index + 1, scanNames.size(), currentScanName)}); Result<> readResults = readData(currentScanName); if(readResults.invalid()) diff --git a/src/Plugins/OrientationAnalysis/test/AlignSectionsMisorientationTest.cpp b/src/Plugins/OrientationAnalysis/test/AlignSectionsMisorientationTest.cpp index 52cc00e757..cf775f76cc 100644 --- a/src/Plugins/OrientationAnalysis/test/AlignSectionsMisorientationTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/AlignSectionsMisorientationTest.cpp @@ -6,8 +6,12 @@ #include "simplnx/Common/Types.hpp" #include "simplnx/Core/Application.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include #include namespace fs = std::filesystem; @@ -25,6 +29,12 @@ using namespace nx::core; TEST_CASE("OrientationAnalysis::AlignSectionsMisorientation Small IN100 Pipeline", "[OrientationAnalysis][AlignSectionsMisorientation]") { UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_misorientation.tar.gz", "align_sections_misorientation"); const nx::core::UnitTest::TestFileSentinel testDataSentinel1(nx::core::unit_test::k_TestFilesDir, "Small_IN100_dream3d_v3.tar.gz", "Small_IN100.dream3d"); @@ -42,6 +52,8 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMisorientation Small IN100 Pipeline // Read the Small IN100 Data set auto baseDataFilePath = fs::path(fmt::format("{}/Small_IN100.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); // MultiThreshold Objects Filter (From SimplnxCore Plugins) SmallIn100::ExecuteMultiThresholdObjects(dataStructure, *filterList); @@ -87,17 +99,24 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMisorientation Small IN100 Pipeline TEST_CASE("OrientationAnalysis::AlignSectionsMisorientationFilter: output test", "[Reconstruction][AlignSectionsMisorientationFilter]") { + UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_misorientation.tar.gz", "align_sections_misorientation"); const nx::core::UnitTest::TestFileSentinel testDataSentinel1(nx::core::unit_test::k_TestFilesDir, "Small_IN100_dream3d_v3.tar.gz", "Small_IN100.dream3d"); - UnitTest::LoadPlugins(); - auto* filterList = Application::Instance()->getFilterList(); // Read the Small IN100 Data set auto baseDataFilePath = fs::path(fmt::format("{}/Small_IN100.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); // MultiThreshold Objects Filter (From SimplnxCore Plugins) SmallIn100::ExecuteMultiThresholdObjects(dataStructure, *filterList); @@ -145,12 +164,18 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMisorientationFilter: output test", const DataPath alignmentAMPath = Constants::k_DataContainerPath.createChildPath(Constants::k_AlignmentAMName); const DataPath slicesPath = alignmentAMPath.createChildPath(Constants::k_SlicesArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(slicesPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(slicesPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(slicesPath), dataStructure.getDataRefAs(slicesPath)); const DataPath relativeShiftsPath = alignmentAMPath.createChildPath(Constants::k_RelativeShiftsArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(relativeShiftsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(relativeShiftsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(relativeShiftsPath), dataStructure.getDataRefAs(relativeShiftsPath)); const DataPath cumulativeShiftsPath = alignmentAMPath.createChildPath(Constants::k_CumulativeShiftsArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(cumulativeShiftsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(cumulativeShiftsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(cumulativeShiftsPath), dataStructure.getDataRefAs(cumulativeShiftsPath)); // Write out the .dream3d file now @@ -160,3 +185,115 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMisorientationFilter: output test", UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); } + +TEST_CASE("OrientationAnalysis::AlignSectionsMisorientation: Benchmark 200x200x200", "[OrientationAnalysis][AlignSectionsMisorientation][.Benchmark]") +{ + UnitTest::LoadPlugins(); + // 200x200x200, Quats float32 4-comp => 200*200*4*4 = 640,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 640000, true); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + + constexpr usize kDimX = 200; + constexpr usize kDimY = 200; + constexpr usize kDimZ = 200; + constexpr usize kSliceVoxels = kDimX * kDimY; + const ShapeType cellTupleShape = {kDimZ, kDimY, kDimX}; + const auto benchmarkFile = fs::path(fmt::format("{}/align_sections_misorientation_benchmark.dream3d", unit_test::k_BinaryTestOutputDir)); + + // Stage 1: Build data programmatically and write to .dream3d + { + DataStructure buildDS; + auto* imageGeom = ImageGeom::Create(buildDS, "DataContainer"); + imageGeom->setDimensions({kDimX, kDimY, kDimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(buildDS, "CellData", cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto* quatsArray = UnitTest::CreateTestDataArray(buildDS, "Quats", cellTupleShape, {4}, cellAM->getId()); + auto& quatsStore = quatsArray->getDataStoreRef(); + auto* phasesArray = UnitTest::CreateTestDataArray(buildDS, "Phases", cellTupleShape, {1}, cellAM->getId()); + auto& phasesStore = phasesArray->getDataStoreRef(); + auto* maskArray = UnitTest::CreateTestDataArray(buildDS, "Mask", cellTupleShape, {1}, cellAM->getId()); + auto& maskStore = maskArray->getDataStoreRef(); + + // Fill using slice-at-a-time bulk writes + constexpr usize kBlockSize = 25; + const float32 cx = kDimX / 2.0f; + const float32 cy = kDimY / 2.0f; + const float32 cz = kDimZ / 2.0f; + const float32 radius = 90.0f; + + std::vector quatsBuf(kSliceVoxels * 4); + std::vector phasesBuf(kSliceVoxels); + std::vector maskBuf(kSliceVoxels); + + for(usize z = 0; z < kDimZ; z++) + { + for(usize y = 0; y < kDimY; y++) + { + for(usize x = 0; x < kDimX; x++) + { + const usize localIdx = y * kDimX + x; + phasesBuf[localIdx] = 1; + + const float32 dx = static_cast(x) - cx; + const float32 dy = static_cast(y) - cy; + const float32 dz = static_cast(z) - cz; + const float32 dist = std::sqrt(dx * dx + dy * dy + dz * dz); + maskBuf[localIdx] = dist < radius ? 1 : 0; + + usize bx = x / kBlockSize; + usize by = y / kBlockSize; + usize bz = z / kBlockSize; + float32 angle = static_cast((bx * 73 + by * 137 + bz * 251) % 360) * (3.14159265f / 180.0f); + float32 halfAngle = angle * 0.5f; + quatsBuf[localIdx * 4 + 0] = std::cos(halfAngle); + quatsBuf[localIdx * 4 + 1] = 0.0f; + quatsBuf[localIdx * 4 + 2] = 0.0f; + quatsBuf[localIdx * 4 + 3] = std::sin(halfAngle); + } + } + quatsStore.copyFromBuffer(z * kSliceVoxels * 4, nonstd::span(quatsBuf.data(), kSliceVoxels * 4)); + phasesStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(phasesBuf.data(), kSliceVoxels)); + maskStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(maskBuf.data(), kSliceVoxels)); + } + + // Create CellEnsembleData with CrystalStructures + const ShapeType ensembleTupleShape = {2}; + auto* ensembleAM = AttributeMatrix::Create(buildDS, "CellEnsembleData", ensembleTupleShape, imageGeom->getId()); + auto* crystalStructsArray = UnitTest::CreateTestDataArray(buildDS, "CrystalStructures", ensembleTupleShape, {1}, ensembleAM->getId()); + auto& crystalStructsStore = crystalStructsArray->getDataStoreRef(); + crystalStructsStore[0] = 999; // Phase 0: Unknown + crystalStructsStore[1] = 1; // Phase 1: Cubic_High + + UnitTest::WriteTestDataStructure(buildDS, benchmarkFile); + } + + // Stage 2: Reload (arrays become OOC-backed) and run filter + DataStructure dataStructure = UnitTest::LoadDataStructure(benchmarkFile); + + { + AlignSectionsMisorientationFilter filter; + Arguments args; + + args.insertOrAssign(AlignSectionsMisorientationFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); + args.insertOrAssign(AlignSectionsMisorientationFilter::k_UseMask_Key, std::make_any(true)); + args.insertOrAssign(AlignSectionsMisorientationFilter::k_MaskArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Mask"}))); + args.insertOrAssign(AlignSectionsMisorientationFilter::k_QuatsArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Quats"}))); + args.insertOrAssign(AlignSectionsMisorientationFilter::k_CellPhasesArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Phases"}))); + args.insertOrAssign(AlignSectionsMisorientationFilter::k_CrystalStructuresArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellEnsembleData", "CrystalStructures"}))); + args.insertOrAssign(AlignSectionsMisorientationFilter::k_SelectedImageGeometryPath_Key, std::make_any(DataPath({"DataContainer"}))); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + fs::remove(benchmarkFile); +} diff --git a/src/Plugins/OrientationAnalysis/test/AlignSectionsMutualInformationTest.cpp b/src/Plugins/OrientationAnalysis/test/AlignSectionsMutualInformationTest.cpp index e8e8f1580e..c4d6ba7ba1 100644 --- a/src/Plugins/OrientationAnalysis/test/AlignSectionsMutualInformationTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/AlignSectionsMutualInformationTest.cpp @@ -3,10 +3,14 @@ #include "OrientationAnalysis/Filters/AlignSectionsMutualInformationFilter.hpp" #include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include #include namespace fs = std::filesystem; @@ -15,12 +19,18 @@ using namespace nx::core; TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: Valid filter execution") { UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_mutual_information.tar.gz", "align_sections_mutual_information"); // We are just going to generate a big number so that we can use that in the output // file path. This tests the creation of intermediate directories that the filter // would be responsible to create. - const uint64_t millisFromEpoch = std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count(); + const uint64 millisFromEpoch = std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count(); auto* filterList = Application::Instance()->getFilterList(); @@ -29,6 +39,8 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: Valid filt // Read Exemplar DREAM3D File Filter auto exemplarFilePath = fs::path(fmt::format("{}/align_sections_mutual_information/6_5_align_sections_mutual_information.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_QuatsArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_QuatsArrayPath)); // Align Sections Mutual Information Filter { @@ -66,11 +78,20 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: Valid filt TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: InValid filter execution") { UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_stats_test_v2.tar.gz", "6_6_stats_test_v2.dream3d"); // Read the Small IN100 Data set auto baseDataFilePath = fs::path(fmt::format("{}/6_6_stats_test_v2.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_QuatsArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_QuatsArrayPath)); + // Instantiate the filter and an Arguments Object AlignSectionsMutualInformationFilter filter; Arguments args; @@ -100,11 +121,20 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: InValid fi TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: output test", "[Reconstruction][AlignSectionsMutualInformationFilter]") { + UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_mutual_information.tar.gz", "align_sections_mutual_information"); // Read Exemplar DREAM3D File Filter auto baseFilePath = fs::path(fmt::format("{}/align_sections_mutual_information/6_5_align_sections_mutual_information.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baseFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_QuatsArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_QuatsArrayPath)); // Align Sections Mutual Information Filter { @@ -143,12 +173,18 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: output tes const DataPath alignmentAMPath = Constants::k_DataContainerPath.createChildPath(Constants::k_AlignmentAMName); const DataPath slicesPath = alignmentAMPath.createChildPath(Constants::k_SlicesArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(slicesPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(slicesPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(slicesPath), dataStructure.getDataRefAs(slicesPath)); const DataPath relativeShiftsPath = alignmentAMPath.createChildPath(Constants::k_RelativeShiftsArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(relativeShiftsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(relativeShiftsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(relativeShiftsPath), dataStructure.getDataRefAs(relativeShiftsPath)); const DataPath cumulativeShiftsPath = alignmentAMPath.createChildPath(Constants::k_CumulativeShiftsArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(cumulativeShiftsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(cumulativeShiftsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(cumulativeShiftsPath), dataStructure.getDataRefAs(cumulativeShiftsPath)); // Write out the .dream3d file now @@ -158,3 +194,115 @@ TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformationFilter: output tes UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +TEST_CASE("OrientationAnalysis::AlignSectionsMutualInformation: Benchmark 200x200x200", "[OrientationAnalysis][AlignSectionsMutualInformationFilter][.Benchmark]") +{ + UnitTest::LoadPlugins(); + // 200x200x200, Quats float32 4-comp => 200*200*4*4 = 640,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 640000, true); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + + constexpr usize kDimX = 200; + constexpr usize kDimY = 200; + constexpr usize kDimZ = 200; + constexpr usize kSliceVoxels = kDimX * kDimY; + const ShapeType cellTupleShape = {kDimZ, kDimY, kDimX}; + const auto benchmarkFile = fs::path(fmt::format("{}/align_sections_mutual_information_benchmark.dream3d", unit_test::k_BinaryTestOutputDir)); + + // Stage 1: Build data programmatically and write to .dream3d + { + DataStructure buildDS; + auto* imageGeom = ImageGeom::Create(buildDS, "DataContainer"); + imageGeom->setDimensions({kDimX, kDimY, kDimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(buildDS, "CellData", cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto* quatsArray = UnitTest::CreateTestDataArray(buildDS, "Quats", cellTupleShape, {4}, cellAM->getId()); + auto& quatsStore = quatsArray->getDataStoreRef(); + auto* phasesArray = UnitTest::CreateTestDataArray(buildDS, "Phases", cellTupleShape, {1}, cellAM->getId()); + auto& phasesStore = phasesArray->getDataStoreRef(); + auto* maskArray = UnitTest::CreateTestDataArray(buildDS, "Mask", cellTupleShape, {1}, cellAM->getId()); + auto& maskStore = maskArray->getDataStoreRef(); + + // Fill using slice-at-a-time bulk writes + constexpr usize kBlockSize = 25; + const float32 cx = kDimX / 2.0f; + const float32 cy = kDimY / 2.0f; + const float32 cz = kDimZ / 2.0f; + const float32 radius = 90.0f; + + std::vector quatsBuf(kSliceVoxels * 4); + std::vector phasesBuf(kSliceVoxels); + std::vector maskBuf(kSliceVoxels); + + for(usize z = 0; z < kDimZ; z++) + { + for(usize y = 0; y < kDimY; y++) + { + for(usize x = 0; x < kDimX; x++) + { + const usize localIdx = y * kDimX + x; + phasesBuf[localIdx] = 1; + + const float32 dx = static_cast(x) - cx; + const float32 dy = static_cast(y) - cy; + const float32 dz = static_cast(z) - cz; + const float32 dist = std::sqrt(dx * dx + dy * dy + dz * dz); + maskBuf[localIdx] = dist < radius ? 1 : 0; + + usize bx = x / kBlockSize; + usize by = y / kBlockSize; + usize bz = z / kBlockSize; + float32 angle = static_cast((bx * 73 + by * 137 + bz * 251) % 360) * (3.14159265f / 180.0f); + float32 halfAngle = angle * 0.5f; + quatsBuf[localIdx * 4 + 0] = std::cos(halfAngle); + quatsBuf[localIdx * 4 + 1] = 0.0f; + quatsBuf[localIdx * 4 + 2] = 0.0f; + quatsBuf[localIdx * 4 + 3] = std::sin(halfAngle); + } + } + quatsStore.copyFromBuffer(z * kSliceVoxels * 4, nonstd::span(quatsBuf.data(), kSliceVoxels * 4)); + phasesStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(phasesBuf.data(), kSliceVoxels)); + maskStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(maskBuf.data(), kSliceVoxels)); + } + + // Create CellEnsembleData with CrystalStructures + const ShapeType ensembleTupleShape = {2}; + auto* ensembleAM = AttributeMatrix::Create(buildDS, "CellEnsembleData", ensembleTupleShape, imageGeom->getId()); + auto* crystalStructsArray = UnitTest::CreateTestDataArray(buildDS, "CrystalStructures", ensembleTupleShape, {1}, ensembleAM->getId()); + auto& crystalStructsStore = crystalStructsArray->getDataStoreRef(); + crystalStructsStore[0] = 999; // Phase 0: Unknown + crystalStructsStore[1] = 1; // Phase 1: Cubic_High + + UnitTest::WriteTestDataStructure(buildDS, benchmarkFile); + } + + // Stage 2: Reload (arrays become OOC-backed) and run filter + DataStructure dataStructure = UnitTest::LoadDataStructure(benchmarkFile); + + { + AlignSectionsMutualInformationFilter filter; + Arguments args; + + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_MisorientationTolerance_Key, std::make_any(5.0f)); + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_UseMask_Key, std::make_any(true)); + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_MaskArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Mask"}))); + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_QuatsArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Quats"}))); + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_CellPhasesArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Phases"}))); + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_CrystalStructuresArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellEnsembleData", "CrystalStructures"}))); + args.insertOrAssign(AlignSectionsMutualInformationFilter::k_SelectedImageGeometryPath_Key, std::make_any(DataPath({"DataContainer"}))); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + fs::remove(benchmarkFile); +} diff --git a/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp b/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp index 9b37c50a4c..f4fc6a0a51 100644 --- a/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp @@ -4,8 +4,14 @@ #include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" #include "OrientationAnalysisTestUtils.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" +#include #include namespace fs = std::filesystem; @@ -31,9 +37,114 @@ const std::string k_CStuctsName = "Crystal Structures"; const DataPath k_CStuctsArrayPath = k_CellEnsembleDataPath.createChildPath(k_CStuctsName); } // namespace VerificationConstants +namespace +{ +constexpr usize k_BenchDim = 200; +constexpr usize k_BenchTotalVoxels = k_BenchDim * k_BenchDim * k_BenchDim; +constexpr usize k_BenchHalf = k_BenchDim / 2; +const float32 k_BaseAngles[8] = {0.0f, 15.0f, 30.0f, 45.0f, 60.0f, 75.0f, 90.0f, 105.0f}; + +void BuildOrientationOctantDataset(DataStructure& ds) +{ + const ShapeType cellTupleShape = {k_BenchDim, k_BenchDim, k_BenchDim}; + + ImageGeom* imageGeom = ImageGeom::Create(ds, VerificationConstants::k_ImageName); + imageGeom->setDimensions({k_BenchDim, k_BenchDim, k_BenchDim}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + AttributeMatrix* cellAM = AttributeMatrix::Create(ds, Constants::k_Cell_Data, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + AttributeMatrix* ensembleAM = AttributeMatrix::Create(ds, Constants::k_Cell_Ensemble_Data, {2}, imageGeom->getId()); + + // Crystal Structures: [999 (Unknown), 1 (Cubic_High)] + auto csStore = DataStoreUtilities::CreateDataStore({2}, {1}, IDataAction::Mode::Execute); + auto* crystalStructures = DataArray::Create(ds, VerificationConstants::k_CStuctsName, csStore, ensembleAM->getId()); + auto& csRef = crystalStructures->getDataStoreRef(); + csRef.setValue(0, 999); + csRef.setValue(1, 1); + + // Phases: all phase 1 — bulk-write per Z-slice to avoid per-element OOC overhead + auto phasesStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* phases = DataArray::Create(ds, VerificationConstants::k_PhasesName, phasesStore, cellAM->getId()); + auto& phasesRef = phases->getDataStoreRef(); + { + const usize sliceSize = k_BenchDim * k_BenchDim; + std::vector phaseSlice(sliceSize, 1); + for(usize iz = 0; iz < k_BenchDim; iz++) + { + phasesRef.copyFromBuffer(iz * sliceSize, nonstd::span(phaseSlice.data(), sliceSize)); + } + } + + // Mask: ~40% bad voxels (mask=0) — bulk-write per Z-slice + auto maskStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* mask = DataArray::Create(ds, VerificationConstants::k_MaskName, maskStore, cellAM->getId()); + auto& maskRef = mask->getDataStoreRef(); + { + const usize sliceSize = k_BenchDim * k_BenchDim; + std::vector maskSlice(sliceSize); + for(usize iz = 0; iz < k_BenchDim; iz++) + { + for(usize iy = 0; iy < k_BenchDim; iy++) + { + for(usize ix = 0; ix < k_BenchDim; ix++) + { + const usize inSlice = iy * k_BenchDim + ix; + const usize globalIdx = iz * sliceSize + inSlice; + maskSlice[inSlice] = static_cast(((globalIdx * 7 + 13) % 100) >= 40 ? 1 : 0); + } + } + maskRef.copyFromBuffer(iz * sliceSize, nonstd::span(maskSlice.data(), sliceSize)); + } + } + + // Quats: 8 octants with distinct base quaternions (15 deg apart about Z axis) — bulk-write per Z-slice + auto quatsStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {4}, IDataAction::Mode::Execute); + auto* quats = DataArray::Create(ds, VerificationConstants::k_QuatsName, quatsStore, cellAM->getId()); + auto& quatsRef = quats->getDataStoreRef(); + { + const usize sliceSize = k_BenchDim * k_BenchDim; + const usize quatSliceElems = sliceSize * 4; + std::vector quatSlice(quatSliceElems); + for(usize iz = 0; iz < k_BenchDim; iz++) + { + for(usize iy = 0; iy < k_BenchDim; iy++) + { + for(usize ix = 0; ix < k_BenchDim; ix++) + { + const usize inSlice = iy * k_BenchDim + ix; + const usize octant = (iz >= k_BenchHalf ? 4 : 0) + (iy >= k_BenchHalf ? 2 : 0) + (ix >= k_BenchHalf ? 1 : 0); + const usize globalIdx = iz * sliceSize + inSlice; + + const float32 perturbDeg = 0.5f * static_cast(globalIdx % 10); + const float32 angleDeg = k_BaseAngles[octant] + perturbDeg; + const float32 halfAngle = angleDeg * 3.14159265f / 360.0f; + + const float32 w = std::cos(halfAngle); + const float32 zVal = std::sin(halfAngle); + quatSlice[inSlice * 4 + 0] = w; + quatSlice[inSlice * 4 + 1] = 0.0f; + quatSlice[inSlice * 4 + 2] = 0.0f; + quatSlice[inSlice * 4 + 3] = zVal; + } + } + quatsRef.copyFromBuffer(iz * quatSliceElems, nonstd::span(quatSlice.data(), quatSliceElems)); + } + } +} +} // namespace + // Case 1.1.1: Base Case | 2 phase | Tolerance 5 | 1 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.1.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_1/case_1_1_1/case_1_1_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -92,6 +203,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.1. // Case 1.1.2: Invalid Base Case | 3 phase | Tolerance 5 | 1 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.1.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_1/case_1_1_2/case_1_1_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -150,6 +267,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.1. // Case 1.1.3: Invalid Base Case | 2 phase | Tolerance 5 | 1 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.1.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_1/case_1_1_3/case_1_1_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -208,6 +331,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.1. // Case 1.2.1: Base Case | 2 phase | Tolerance 5 | 2 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.2.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_2/case_1_2_1/case_1_2_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -266,6 +395,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.2. // Case 1.2.2: Invalid Base Case | 2 phase | Tolerance 5 | 2 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.2.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_2/case_1_2_2/case_1_2_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -324,6 +459,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.2. // Case 1.2.3: Invalid Base Case | 2 phase | Tolerance 5 | 2 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.2.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_2/case_1_2_3/case_1_2_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -382,6 +523,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.2. // Case 1.3.1: Base Case | 1 phase | Tolerance 5 | 3 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.3.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_3/case_1_3_1/case_1_3_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -440,6 +587,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.3. // Case 1.3.2: Invalid Base Case | 2 phase | Tolerance 5 | 3 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.3.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_3/case_1_3_2/case_1_3_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -498,6 +651,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.3. // Case 1.3.3: Invalid Base Case | 1 phase | Tolerance 5 | 3 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.3.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_3/case_1_3_3/case_1_3_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -556,6 +715,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.3. // Case 1.4.1: Base Case | 1 phase | Tolerance 5 | 4 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.4.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_4/case_1_4_1/case_1_4_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -614,6 +779,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.4. // Case 1.4.2: Invalid Base Case | 2 phase | Tolerance 5 | 4 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.4.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_4/case_1_4_2/case_1_4_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -672,6 +843,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.4. // Case 1.4.3: Invalid Base Case | 1 phase | Tolerance 5 | 4 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.4.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_4/case_1_4_3/case_1_4_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -730,6 +907,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.4. // Case 1.5.1: Base Case | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.5.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_5/case_1_5_1/case_1_5_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -788,6 +971,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.5. // Case 1.5.2: Invalid Base Case | 2 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.5.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_5/case_1_5_2/case_1_5_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -846,6 +1035,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.5. // Case 1.5.3: Invalid Base Case | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.5.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_5/case_1_5_3/case_1_5_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -904,6 +1099,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.5. // Case 1.6.1: Base Case | 1 phase | Tolerance 5 | 6 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.6.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_6/case_1_6_1/case_1_6_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -962,6 +1163,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.6. // Case 1.6.2: Invalid Base Case | 2 phase | Tolerance 5 | 6 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.6.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_6/case_1_6_2/case_1_6_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -1020,6 +1227,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.6. // Case 1.6.3: Invalid Base Case | 1 phase | Tolerance 5 | 6 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.6.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 140, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_1/case_1_6/case_1_6_3/case_1_6_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -1078,6 +1291,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 1.6. // Case 2.1: X+ Dim Case (Sequential) | Valid | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_2/case_2_1/case_2_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -1132,6 +1351,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.1" // Case 2.2: Y+ Dim Case (Sequential) | Valid | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_2/case_2_2/case_2_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -1186,6 +1411,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.2" // Case 2.3: Z+ Dim Case (Sequential) | Valid | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.3", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_2/case_2_3/case_2_3_input.dream3d", unit_test::k_TestFilesDir)); @@ -1240,6 +1471,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.3" // Case 2.4: X- Dim Case (Recursive) | Valid | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.4", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_2/case_2_4/case_2_4_input.dream3d", unit_test::k_TestFilesDir)); @@ -1294,6 +1531,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.4" // Case 2.5: Y- Dim Case (Recursive) | Valid | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.5", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_2/case_2_5/case_2_5_input.dream3d", unit_test::k_TestFilesDir)); @@ -1348,6 +1591,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.5" // Case 2.6: Z- Dim Case (Recursive) | Valid | 1 phase | Tolerance 5 | 5 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.6", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_2/case_2_6/case_2_6_input.dream3d", unit_test::k_TestFilesDir)); @@ -1402,6 +1651,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 2.6" // Case 3.1: Long Sequential | Valid | 1 phase | Tolerance 5 | 1 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 3.1", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_3/case_3_1/case_3_1_input.dream3d", unit_test::k_TestFilesDir)); @@ -1456,6 +1711,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 3.1" // Case 3.2: Long Recursive | Valid | 1 phase | Tolerance 5 | 1 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 3.2", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_3/case_3_2/case_3_2_input.dream3d", unit_test::k_TestFilesDir)); @@ -1510,6 +1771,12 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 3.2" // Case 4: Semi-Complex Synthetic Structure | Valid | 3 phase | Tolerance 5 | 4 Min Neighbors TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 4", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") { + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "7_bad_data_neighbor_orientation_check.tar.gz", "bad_data_neighbor_orientation_check"); auto baseDataFilePath = fs::path(fmt::format("{}/bad_data_neighbor_orientation_check/case_4/case_4_input.dream3d", unit_test::k_TestFilesDir)); @@ -1599,3 +1866,72 @@ TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Case 4", UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: Generate Large Test Dataset", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); + + DataStructure buildDS; + BuildOrientationOctantDataset(buildDS); + + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data"; + fs::create_directories(outputDir); + const auto outputFile = outputDir / "bad_data_neighbor_orientation_check_data.dream3d"; + UnitTest::WriteTestDataStructure(buildDS, outputFile); + fmt::print("Wrote test data to: {}\n", outputFile.string()); +} + +TEST_CASE("OrientationAnalysis::BadDataNeighborOrientationCheckFilter: 200x200x200 octant orientations", "[OrientationAnalysis][BadDataNeighborOrientationCheckFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 10000, true); + + DataStructure dataStructure; + BuildOrientationOctantDataset(dataStructure); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(VerificationConstants::k_QuatsArrayPath)); + + // Count initial bad voxels + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(VerificationConstants::k_MaskArrayPath)); + const auto& maskBefore = dataStructure.getDataRefAs>(VerificationConstants::k_MaskArrayPath).getDataStoreRef(); + usize badBefore = 0; + for(usize i = 0; i < k_BenchTotalVoxels; i++) + { + if(maskBefore[i] == 0) + { + badBefore++; + } + } + + { + BadDataNeighborOrientationCheckFilter filter; + Arguments args; + + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_MisorientationTolerance_Key, std::make_any(5.0f)); + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_NumberOfNeighbors_Key, std::make_any(1)); + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_ImageGeometryPath_Key, std::make_any(VerificationConstants::k_ImagePath)); + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_MaskArrayPath_Key, std::make_any(VerificationConstants::k_MaskArrayPath)); + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_CellPhasesArrayPath_Key, std::make_any(VerificationConstants::k_PhasesArrayPath)); + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_QuatsArrayPath_Key, std::make_any(VerificationConstants::k_QuatsArrayPath)); + args.insertOrAssign(BadDataNeighborOrientationCheckFilter::k_CrystalStructuresArrayPath_Key, std::make_any(VerificationConstants::k_CStuctsArrayPath)); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + // Verify: Some bad voxels should have been corrected (mask flipped 0->1) + const auto& maskAfter = dataStructure.getDataRefAs>(VerificationConstants::k_MaskArrayPath).getDataStoreRef(); + usize badAfter = 0; + for(usize i = 0; i < k_BenchTotalVoxels; i++) + { + if(maskAfter[i] == 0) + { + badAfter++; + } + } + REQUIRE(badAfter < badBefore); +} diff --git a/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp b/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp index 5a6e1a4681..d806804c4b 100644 --- a/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp @@ -2,257 +2,383 @@ #include "OrientationAnalysis/Filters/CAxisSegmentFeaturesFilter.hpp" #include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" -#include "OrientationAnalysisTestUtils.hpp" -#include "simplnx/Core/Application.hpp" -#include "simplnx/Parameters/ArrayCreationParameter.hpp" -#include "simplnx/Parameters/Dream3dImportParameter.hpp" -#include "simplnx/Parameters/GeometrySelectionParameter.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Parameters/ChoicesParameter.hpp" +#include "simplnx/UnitTest/SegmentFeaturesTestUtils.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" -#include +#include namespace fs = std::filesystem; using namespace nx::core; -using namespace nx::core::Constants; -namespace caxis_segment_features_constants -{ -inline constexpr StringLiteral k_InputGeometryName = "DataContainer"; -inline const DataPath k_InputGeometryPath({k_InputGeometryName}); -inline constexpr StringLiteral k_CellDataName = "CellData"; -inline constexpr StringLiteral k_EnsembleName = "CellEnsembleData"; -inline const DataPath k_QuatsArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Quats"); -inline const DataPath k_PhasesArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Phases"); -inline const DataPath k_MaskArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Mask (Y Pos)"); - -inline const DataPath k_CrystalStructuresArrayPath = k_InputGeometryPath.createChildPath(k_EnsembleName).createChildPath("CrystalStructures"); - -inline const DataPath k_ActivesArrayPath = k_InputGeometryPath.createChildPath(k_Grain_Data).createChildPath(k_ActiveName); - -inline const DataPath k_FeatureIdsArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath(k_FeatureIds); - -inline const DataPath k_FeatureIdsFacePath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("CAxis_FeatureIds_Face"); -inline const DataPath k_FeatureIdsAllPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("CAxis_FeatureIds_All"); -inline const DataPath k_FeatureIdsMaskFacePath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("CAxis_FeatureIds_Mask_Face"); -inline const DataPath k_FeatureIdsMaskAllPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("CAxis_FeatureIds_Mask_All"); -} // namespace caxis_segment_features_constants +using namespace nx::core::UnitTest; -TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures:Face", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +namespace { - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); +// Exemplar archive (shared across Scalar, EBSD, CAxis) +const std::string k_ArchiveName = "segment_features_exemplars.tar.gz"; +const std::string k_DataDirName = "segment_features_exemplars"; +const fs::path k_DataDir = fs::path(unit_test::k_TestFilesDir.view()) / k_DataDirName; +const fs::path k_SmallExemplarFile = k_DataDir / "caxis_small.dream3d"; +const fs::path k_LargeExemplarFile = k_DataDir / "caxis_large.dream3d"; + +// Geometry names +constexpr StringLiteral k_GeomName = "DataContainer"; +constexpr StringLiteral k_CellDataName = "CellData"; +constexpr StringLiteral k_FeatureDataName = "CellFeatureData"; +constexpr StringLiteral k_EnsembleName = "CellEnsembleData"; + +// Output array paths +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_FeatureIdsPath({k_GeomName, k_CellDataName, "FeatureIds"}); +const DataPath k_ActivePath({k_GeomName, k_FeatureDataName, "Active"}); +const DataPath k_MaskPath({k_GeomName, k_CellDataName, "Mask"}); +const DataPath k_QuatsPath({k_GeomName, k_CellDataName, "Quats"}); +const DataPath k_PhasesPath({k_GeomName, k_CellDataName, "Phases"}); +const DataPath k_CrystalStructuresPath({k_GeomName, k_EnsembleName, "CrystalStructures"}); + +// Test dimensions +constexpr usize k_SmallDim = 15; +constexpr usize k_SmallBlockSize = 5; +constexpr usize k_LargeDim = 200; +constexpr usize k_LargeBlockSize = 25; + +/** + * @brief Populates CAxisSegmentFeaturesFilter arguments. + */ +void SetupArgs(Arguments& args, bool useMask, float32 tolerance = 5.0f, ChoicesParameter::ValueType neighborScheme = 0, bool randomize = false) +{ + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(tolerance)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborScheme)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_UseMask_Key, std::make_any(useMask)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(useMask ? k_MaskPath : DataPath{})); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(k_QuatsPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(k_PhasesPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(k_CrystalStructuresPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(std::string(k_FeatureDataName))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(randomize)); +} +} // namespace - // EBSD Segment Features/Semgent Features (Misorientation) Filter +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: Small Correctness", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // Quats float32 4-comp => 15*15*4*4 = 3,600 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 3600, true); + + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_SmallExemplarFile); + + std::string testName = GENERATE("Base", "Masked"); + DYNAMIC_SECTION("Variant: " << testName) { - CAxisSegmentFeaturesFilter filter; - Arguments args; + const bool useMask = (testName == "Masked"); + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; - // Create default Parameters for the filter. - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(0)); + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 0, k_SmallBlockSize); // Hexagonal_High - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(caxis_segment_features_constants::k_MaskArrayPath)); + if(useMask) + { + BuildSphericalMask(dataStructure, cellShape, am->getId()); + } - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(caxis_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(caxis_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(caxis_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(caxis_segment_features_constants::k_CrystalStructuresArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({k_GeomName, k_CellDataName, "Quats"}))); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + CAxisSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, useMask); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } - { - UInt8Array& actives = dataStructure.getDataRefAs(caxis_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 57); - } + // Compare against exemplar + const std::string exemplarGeomName = testName + "_Exemplar"; + const DataPath exemplarFeatureIdsPath({exemplarGeomName, std::string(k_CellDataName), "FeatureIds"}); + const DataPath exemplarActivePath({exemplarGeomName, std::string(k_FeatureDataName), "Active"}); - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group - { - const auto& generatedDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsFacePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(k_FeatureIdsPath)); - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); - } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarActivePath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarActivePath), dataStructure.getDataRefAs(k_ActivePath)); - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } } -TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures:All", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: 200x200x200 Large OOC", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") { - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // Quats float32 4-comp => 200*200*4*4 = 640,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 640000, true); - // EBSD Segment Features/Semgent Features (Misorientation) Filter - { - CAxisSegmentFeaturesFilter filter; - Arguments args; + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_LargeExemplarFile); - // Create default Parameters for the filter. - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(1)); + const ShapeType cellShape = {k_LargeDim, k_LargeDim, k_LargeDim}; + const std::array dims = {k_LargeDim, k_LargeDim, k_LargeDim}; - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(caxis_segment_features_constants::k_MaskArrayPath)); + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 0, k_LargeBlockSize); // Hexagonal_High + BuildSphericalMask(dataStructure, cellShape, am->getId()); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(caxis_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(caxis_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(caxis_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(caxis_segment_features_constants::k_CrystalStructuresArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({k_GeomName, k_CellDataName, "Quats"}))); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + CAxisSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/true); - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } + const DataPath exemplarFeatureIdsPath({"DataContainer_Exemplar", std::string(k_CellDataName), "FeatureIds"}); + const DataPath exemplarActivePath({"DataContainer_Exemplar", std::string(k_FeatureDataName), "Active"}); - { - UInt8Array& actives = dataStructure.getDataRefAs(caxis_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 37); - } - - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group - { - const auto& generatedDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsAllPath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(k_FeatureIdsPath)); - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); - } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarActivePath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarActivePath), dataStructure.getDataRefAs(k_ActivePath)); - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); + UnitTest::CheckArraysInheritTupleDims(dataStructure); } -TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures:MaskFace", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: No Valid Voxels Returns Error", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") { - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + UnitTest::LoadPlugins(); - // EBSD Segment Features/Semgent Features (Misorientation) Filter - { - CAxisSegmentFeaturesFilter filter; - Arguments args; + RunNoValidVoxelsErrorTest([](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, const DataPath& maskPath) { + const ShapeType cellShape = {3, 3, 3}; + auto& am = ds.getDataRefAs(cellDataPath); + auto& geom = ds.getDataRefAs(geomPath); + BuildOrientationTestData(ds, cellShape, geom.getId(), am.getId(), 0, 3); // Hexagonal_High - // Create default Parameters for the filter. args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); args.insertOrAssign(CAxisSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(0)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_UseMask_Key, std::make_any(true)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(caxis_segment_features_constants::k_MaskArrayPath)); - - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(caxis_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(caxis_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(caxis_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(caxis_segment_features_constants::k_CrystalStructuresArrayPath)); - - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(maskPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(geomPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(cellDataPath.createChildPath("Quats"))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(cellDataPath.createChildPath("Phases"))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(DataPath({"Geom", "CellEnsembleData", "CrystalStructures"}))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any("Grain Data")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); args.insertOrAssign(CAxisSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + }); +} - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } - +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: Randomize Feature IDs", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +{ + UnitTest::LoadPlugins(); + + constexpr usize k_ExpectedFeatures = 3; // 3 Z-layers with 1 merge-pair pillar + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; + + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 0, k_SmallBlockSize); // Hexagonal_High + + CAxisSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/false, /*tolerance=*/5.0f, /*neighborScheme=*/0, /*randomize=*/true); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + const auto& actives = dataStructure.getDataRefAs(k_ActivePath); + REQUIRE(actives.getNumberOfTuples() == k_ExpectedFeatures + 1); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + std::set uniqueIds; + int32 minId = std::numeric_limits::max(); + int32 maxId = std::numeric_limits::min(); + for(usize i = 0; i < featureStore.getNumberOfTuples(); i++) { - UInt8Array& actives = dataStructure.getDataRefAs(caxis_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 31); + int32 fid = featureStore.getValue(i); + uniqueIds.insert(fid); + minId = std::min(minId, fid); + maxId = std::max(maxId, fid); } + REQUIRE(minId == 1); + REQUIRE(maxId == static_cast(k_ExpectedFeatures)); + REQUIRE(uniqueIds.size() == k_ExpectedFeatures); +} - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: High Tolerance Merges All", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +{ + UnitTest::LoadPlugins(); + + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; + + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 0, k_SmallBlockSize); // Hexagonal_High + + CAxisSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/false, /*tolerance=*/90.0f); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // With tolerance=90 degrees, all C-axis directions on the hemisphere merge into 1 feature + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + const auto& actives = dataStructure.getDataRefAs(k_ActivePath); + REQUIRE(actives.getNumberOfTuples() == 2); // 1 feature + index 0 + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + for(usize i = 0; i < featureStore.getNumberOfTuples(); i++) { - const auto& generatedDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsMaskFacePath); - - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); + REQUIRE(featureStore.getValue(i) == 1); } - - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); } -TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures:MaskAll", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: FaceEdgeVertex Connectivity", "[OrientationAnalysis][CAxisSegmentFeaturesFilter]") { - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + UnitTest::LoadPlugins(); + + constexpr float32 k_DegToRad = 3.14159265358979323846f / 180.0f; + + auto setupCAxis = [&](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, ChoicesParameter::ValueType neighborScheme) { + const ShapeType cellShape = {3, 3, 3}; + auto& am = ds.getDataRefAs(cellDataPath); + auto& geom = ds.getDataRefAs(geomPath); + + // Quaternions: background = 60° X-rotation, pairs = identity and 30° (EBSDlib order: x,y,z,w) + const float32 bgHalf = 60.0f * k_DegToRad * 0.5f; + auto quatsDS = DataStoreUtilities::CreateDataStore(cellShape, {4}, IDataAction::Mode::Execute); + auto* quatsArr = DataArray::Create(ds, "Quats", quatsDS, am.getId()); + auto& quatsStore = quatsArr->getDataStoreRef(); + for(usize i = 0; i < 27; i++) + { + quatsStore[i * 4 + 0] = std::sin(bgHalf); + quatsStore[i * 4 + 1] = 0.0f; + quatsStore[i * 4 + 2] = 0.0f; + quatsStore[i * 4 + 3] = std::cos(bgHalf); + } + for(usize idx : {static_cast(0), static_cast(1 * 9 + 1 * 3 + 1)}) + { + quatsStore[idx * 4 + 0] = 0.0f; + quatsStore[idx * 4 + 1] = 0.0f; + quatsStore[idx * 4 + 2] = 0.0f; + quatsStore[idx * 4 + 3] = 1.0f; + } + const float32 pairHalf = 30.0f * k_DegToRad * 0.5f; + for(usize idx : {static_cast(0 * 9 + 0 * 3 + 2), static_cast(1 * 9 + 1 * 3 + 2)}) + { + quatsStore[idx * 4 + 0] = std::sin(pairHalf); + quatsStore[idx * 4 + 1] = 0.0f; + quatsStore[idx * 4 + 2] = 0.0f; + quatsStore[idx * 4 + 3] = std::cos(pairHalf); + } + + auto phasesDS = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* phasesArr = DataArray::Create(ds, "Phases", phasesDS, am.getId()); + phasesArr->fill(1); + + const ShapeType ensShape = {2}; + auto* ensAM = AttributeMatrix::Create(ds, "CellEnsembleData", ensShape, geom.getId()); + auto crystDS = DataStoreUtilities::CreateDataStore(ensShape, {1}, IDataAction::Mode::Execute); + auto* crystArr = DataArray::Create(ds, "CrystalStructures", crystDS, ensAM->getId()); + auto& crystStore = crystArr->getDataStoreRef(); + crystStore[0] = 999; + crystStore[1] = 0; // Hexagonal_High + + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0f)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborScheme)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(DataPath{})); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(geomPath)); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(cellDataPath.createChildPath("Quats"))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(cellDataPath.createChildPath("Phases"))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(DataPath({"Geom", "CellEnsembleData", "CrystalStructures"}))); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any("CellFeatureData")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(CAxisSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + }; - // EBSD Segment Features/Semgent Features (Misorientation) Filter - { - CAxisSegmentFeaturesFilter filter; - Arguments args; + RunFaceEdgeVertexConnectivityTest([&](Arguments& args, DataStructure& ds, const DataPath& gp, const DataPath& cp) { setupCAxis(args, ds, gp, cp, 0); }, + [&](Arguments& args, DataStructure& ds, const DataPath& gp, const DataPath& cp) { setupCAxis(args, ds, gp, cp, 1); }); +} - // Create default Parameters for the filter. - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(1)); +TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: Generate Test Data", "[OrientationAnalysis][CAxisSegmentFeaturesFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_UseMask_Key, std::make_any(true)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(caxis_segment_features_constants::k_MaskArrayPath)); + const auto outputDir = fs::path(fmt::format("{}/generated_test_data/caxis_segment_features", unit_test::k_BinaryTestOutputDir)); + fs::create_directories(outputDir); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(caxis_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(caxis_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(caxis_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(caxis_segment_features_constants::k_CrystalStructuresArrayPath)); + // Small input data (15^3) — one geometry per test variant + { + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(CAxisSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + DataStructure ds; - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto* amBase = BuildSegmentFeaturesTestGeometry(ds, dims, "Base", std::string(k_CellDataName)); + auto& geomBase = ds.getDataRefAs(DataPath({"Base"})); + BuildOrientationTestData(ds, cellShape, geomBase.getId(), amBase->getId(), 0, k_SmallBlockSize); // Hexagonal_High - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } + auto* amMasked = BuildSegmentFeaturesTestGeometry(ds, dims, "Masked", std::string(k_CellDataName)); + auto& geomMasked = ds.getDataRefAs(DataPath({"Masked"})); + BuildOrientationTestData(ds, cellShape, geomMasked.getId(), amMasked->getId(), 0, k_SmallBlockSize); + BuildSphericalMask(ds, cellShape, amMasked->getId()); - { - UInt8Array& actives = dataStructure.getDataRefAs(caxis_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 25); + UnitTest::WriteTestDataStructure(ds, outputDir / "small_input.dream3d"); } - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group + // Large input data (200^3) — mask=true { - const auto& generatedDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(caxis_segment_features_constants::k_FeatureIdsMaskAllPath); + const ShapeType cellShape = {k_LargeDim, k_LargeDim, k_LargeDim}; + const std::array dims = {k_LargeDim, k_LargeDim, k_LargeDim}; - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); - } + DataStructure ds; + auto* am = BuildSegmentFeaturesTestGeometry(ds, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = ds.getDataRefAs(k_GeomPath); + BuildOrientationTestData(ds, cellShape, geom.getId(), am->getId(), 0, k_LargeBlockSize); // Hexagonal_High + BuildSphericalMask(ds, cellShape, am->getId()); - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); + UnitTest::WriteTestDataStructure(ds, outputDir / "large_input.dream3d"); + } } diff --git a/src/Plugins/OrientationAnalysis/test/ComputeGBCDPoleFigureTest.cpp b/src/Plugins/OrientationAnalysis/test/ComputeGBCDPoleFigureTest.cpp index 5b9edc5828..6ed1ecbae4 100644 --- a/src/Plugins/OrientationAnalysis/test/ComputeGBCDPoleFigureTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ComputeGBCDPoleFigureTest.cpp @@ -7,6 +7,7 @@ #include "simplnx/Parameters/NumberParameter.hpp" #include "simplnx/Parameters/VectorParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include @@ -37,6 +38,9 @@ TEST_CASE("OrientationAnalysis::ComputeGBCDPoleFigureFilter", "[OrientationAnaly { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_Small_IN100_GBCD.tar.gz", "6_6_Small_IN100_GBCD"); // Read the Small IN100 Data set diff --git a/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp b/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp index 8e4e937797..ce569f0114 100644 --- a/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ComputeIPFColorsTest.cpp @@ -24,6 +24,7 @@ Compare the data sets. The values should be exactly the same. #include "simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp" #include "simplnx/Parameters/VectorParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" #include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" @@ -52,6 +53,8 @@ constexpr StringLiteral k_OutputIPFColors("IPF Colors_Test_Output"); TEST_CASE("OrientationAnalysis::ComputeIPFColors", "[OrientationAnalysis][ComputeIPFColorsFilter]") { + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); { HomochoricDType ho(0.021797740480252403, 0.027063934475102136, 0.035554288118377242); diff --git a/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp b/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp index 3765e2f6f1..bff8e2b2e8 100644 --- a/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp @@ -2,268 +2,397 @@ #include "OrientationAnalysis/Filters/EBSDSegmentFeaturesFilter.hpp" #include "OrientationAnalysis/OrientationAnalysis_test_dirs.hpp" -#include "OrientationAnalysisTestUtils.hpp" -#include "simplnx/Core/Application.hpp" -#include "simplnx/Parameters/ArrayCreationParameter.hpp" -#include "simplnx/Parameters/Dream3dImportParameter.hpp" -#include "simplnx/Parameters/GeometrySelectionParameter.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Parameters/ChoicesParameter.hpp" +#include "simplnx/UnitTest/SegmentFeaturesTestUtils.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" -#include - -#include +#include namespace fs = std::filesystem; using namespace nx::core; -using namespace nx::core::Constants; +using namespace nx::core::UnitTest; -namespace ebsd_segment_features_constants +namespace { -inline constexpr StringLiteral k_InputGeometryName = "DataContainer"; -inline const DataPath k_InputGeometryPath({k_InputGeometryName}); -inline constexpr StringLiteral k_CellDataName = "CellData"; -inline constexpr StringLiteral k_EnsembleName = "CellEnsembleData"; -inline const DataPath k_QuatsArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Quats"); -inline const DataPath k_PhasesArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Phases"); -inline const DataPath k_MaskArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Mask (Y Pos)"); - -inline const DataPath k_CrystalStructuresArrayPath = k_InputGeometryPath.createChildPath(k_EnsembleName).createChildPath("CrystalStructures"); - -inline const DataPath k_ActivesArrayPath = k_InputGeometryPath.createChildPath(k_Grain_Data).createChildPath(k_ActiveName); - -inline const DataPath k_FeatureIdsArrayPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath(k_FeatureIds); - -inline const DataPath k_FeatureIdsFacePath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Ebsd_FeatureIds_Face"); -inline const DataPath k_FeatureIdsAllPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Ebsd_FeatureIds_All"); -inline const DataPath k_FeatureIdsMaskFacePath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Ebsd_FeatureIds_Mask_Face"); -inline const DataPath k_FeatureIdsMaskAllPath = k_InputGeometryPath.createChildPath(k_CellDataName).createChildPath("Ebsd_FeatureIds_Mask_All"); -} // namespace ebsd_segment_features_constants +// Exemplar archive (shared across Scalar, EBSD, CAxis) +const std::string k_ArchiveName = "segment_features_exemplars.tar.gz"; +const std::string k_DataDirName = "segment_features_exemplars"; +const fs::path k_DataDir = fs::path(unit_test::k_TestFilesDir.view()) / k_DataDirName; +const fs::path k_SmallExemplarFile = k_DataDir / "ebsd_small.dream3d"; +const fs::path k_LargeExemplarFile = k_DataDir / "ebsd_large.dream3d"; + +// Geometry names +constexpr StringLiteral k_GeomName = "DataContainer"; +constexpr StringLiteral k_CellDataName = "CellData"; +constexpr StringLiteral k_FeatureDataName = "CellFeatureData"; +constexpr StringLiteral k_EnsembleName = "CellEnsembleData"; + +// Output array paths +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_FeatureIdsPath({k_GeomName, k_CellDataName, "FeatureIds"}); +const DataPath k_ActivePath({k_GeomName, k_FeatureDataName, "Active"}); +const DataPath k_MaskPath({k_GeomName, k_CellDataName, "Mask"}); +const DataPath k_QuatsPath({k_GeomName, k_CellDataName, "Quats"}); +const DataPath k_PhasesPath({k_GeomName, k_CellDataName, "Phases"}); +const DataPath k_CrystalStructuresPath({k_GeomName, k_EnsembleName, "CrystalStructures"}); + +// Test dimensions +constexpr usize k_SmallDim = 15; +constexpr usize k_SmallBlockSize = 5; +constexpr usize k_LargeDim = 200; +constexpr usize k_LargeBlockSize = 25; + +/** + * @brief Populates EBSDSegmentFeaturesFilter arguments. + */ +void SetupArgs(Arguments& args, bool useMask, bool isPeriodic = false, float32 tolerance = 5.0f, ChoicesParameter::ValueType neighborScheme = 0, bool randomize = false) +{ + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(tolerance)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborScheme)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(useMask)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(useMask ? k_MaskPath : DataPath{})); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_IsPeriodic_Key, std::make_any(isPeriodic)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(k_QuatsPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(k_PhasesPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(k_CrystalStructuresPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(std::string(k_FeatureDataName))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(randomize)); +} +} // namespace -TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures:Face", "[OrientationAnalysis][EBSDSegmentFeatures]") +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: Small Correctness", "[OrientationAnalysis][EBSDSegmentFeatures]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // Quats float32 4-comp => 15*15*4*4 = 3,600 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 3600, true); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_SmallExemplarFile); - // EBSD Segment Features/Semgent Features (Misorientation) Filter + std::string testName = GENERATE("Base", "Masked", "Periodic"); + DYNAMIC_SECTION("Variant: " << testName) { - EBSDSegmentFeaturesFilter filter; - Arguments args; + const bool useMask = (testName == "Masked"); + const bool isPeriodic = (testName == "Periodic"); + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; - // Create default Parameters for the filter. - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(0)); + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 1, k_SmallBlockSize, isPeriodic); // Cubic_High - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_MaskArrayPath)); + if(useMask) + { + BuildSphericalMask(dataStructure, cellShape, am->getId()); + } - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(ebsd_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_CrystalStructuresArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({k_GeomName, k_CellDataName, "Quats"}))); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + EBSDSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, useMask, isPeriodic); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } - { - UInt8Array& actives = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 83); - } + // Compare against exemplar + const std::string exemplarGeomName = testName + "_Exemplar"; + const DataPath exemplarFeatureIdsPath({exemplarGeomName, std::string(k_CellDataName), "FeatureIds"}); + const DataPath exemplarActivePath({exemplarGeomName, std::string(k_FeatureDataName), "Active"}); - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group - { - const auto& generatedDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsFacePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(k_FeatureIdsPath)); - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); - } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarActivePath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarActivePath), dataStructure.getDataRefAs(k_ActivePath)); - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } } -TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures:All", "[OrientationAnalysis][EBSDSegmentFeatures]") +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: 200x200x200 Large OOC", "[OrientationAnalysis][EBSDSegmentFeatures]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // Quats float32 4-comp => 200*200*4*4 = 640,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 640000, true); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_LargeExemplarFile); - // EBSD Segment Features/Semgent Features (Misorientation) Filter - { - EBSDSegmentFeaturesFilter filter; - Arguments args; + const ShapeType cellShape = {k_LargeDim, k_LargeDim, k_LargeDim}; + const std::array dims = {k_LargeDim, k_LargeDim, k_LargeDim}; - // Create default Parameters for the filter. - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(1)); + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 1, k_LargeBlockSize, true); // Cubic_High, wrapBoundary - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_MaskArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({k_GeomName, k_CellDataName, "Quats"}))); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(ebsd_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_CrystalStructuresArrayPath)); + EBSDSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/false, /*isPeriodic=*/true); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + const DataPath exemplarFeatureIdsPath({"DataContainer_Exemplar", std::string(k_CellDataName), "FeatureIds"}); + const DataPath exemplarActivePath({"DataContainer_Exemplar", std::string(k_FeatureDataName), "Active"}); - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(k_FeatureIdsPath)); - { - UInt8Array& actives = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 77); - } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarActivePath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarActivePath), dataStructure.getDataRefAs(k_ActivePath)); - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group - { - const auto& generatedDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsAllPath); + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); - } +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: No Valid Voxels Returns Error", "[OrientationAnalysis][EBSDSegmentFeatures]") +{ + UnitTest::LoadPlugins(); + + RunNoValidVoxelsErrorTest([](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, const DataPath& maskPath) { + const ShapeType cellShape = {3, 3, 3}; + auto& am = ds.getDataRefAs(cellDataPath); + auto& geom = ds.getDataRefAs(geomPath); + BuildOrientationTestData(ds, cellShape, geom.getId(), am.getId(), 1, 3); // Cubic_High - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(0)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(true)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(maskPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_IsPeriodic_Key, std::make_any(false)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(geomPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(cellDataPath.createChildPath("Quats"))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(cellDataPath.createChildPath("Phases"))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(DataPath({"Geom", "CellEnsembleData", "CrystalStructures"}))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any("Grain Data")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + }); } -TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures:MaskFace", "[OrientationAnalysis][EBSDSegmentFeatures]") +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: Randomize Feature IDs", "[OrientationAnalysis][EBSDSegmentFeatures]") { UnitTest::LoadPlugins(); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); - - // EBSD Segment Features/Semgent Features (Misorientation) Filter + constexpr usize k_ExpectedFeatures = 3; // 3 Z-layers with 1 merge-pair pillar + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; + + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 1, k_SmallBlockSize); // Cubic_High + + EBSDSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/false, /*isPeriodic=*/false, /*tolerance=*/5.0f, /*neighborScheme=*/0, /*randomize=*/true); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + const auto& actives = dataStructure.getDataRefAs(k_ActivePath); + REQUIRE(actives.getNumberOfTuples() == k_ExpectedFeatures + 1); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + std::set uniqueIds; + int32 minId = std::numeric_limits::max(); + int32 maxId = std::numeric_limits::min(); + for(usize i = 0; i < featureStore.getNumberOfTuples(); i++) { - EBSDSegmentFeaturesFilter filter; - Arguments args; + int32 fid = featureStore.getValue(i); + uniqueIds.insert(fid); + minId = std::min(minId, fid); + maxId = std::max(maxId, fid); + } + REQUIRE(minId == 1); + REQUIRE(maxId == static_cast(k_ExpectedFeatures)); + REQUIRE(uniqueIds.size() == k_ExpectedFeatures); +} - // Create default Parameters for the filter. - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(0)); +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: High Tolerance Merges All", "[OrientationAnalysis][EBSDSegmentFeatures]") +{ + UnitTest::LoadPlugins(); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(true)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_MaskArrayPath)); + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(ebsd_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_CrystalStructuresArrayPath)); + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = dataStructure.getDataRefAs(k_GeomPath); + BuildOrientationTestData(dataStructure, cellShape, geom.getId(), am->getId(), 1, k_SmallBlockSize); // Cubic_High - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + EBSDSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/false, /*isPeriodic=*/false, /*tolerance=*/90.0f); - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - } + // With tolerance=90 degrees, all orientations merge (max cubic misorientation is ~62.8 deg) + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + const auto& actives = dataStructure.getDataRefAs(k_ActivePath); + REQUIRE(actives.getNumberOfTuples() == 2); // 1 feature + index 0 + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + for(usize i = 0; i < featureStore.getNumberOfTuples(); i++) { - UInt8Array& actives = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 36); + REQUIRE(featureStore.getValue(i) == 1); } +} - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group - { - const auto& generatedDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsMaskFacePath); +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: FaceEdgeVertex Connectivity", "[OrientationAnalysis][EBSDSegmentFeatures]") +{ + UnitTest::LoadPlugins(); - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); - } + // Shared test: verifies vertex and edge connectivity with FaceEdgeVertex scheme. + // Setup lambda creates orientation data with 4 isolated voxels and configures args. + // Pair voxels share the same quaternion (0° X-rotation = identity). + // Background voxels get a different quaternion (60° X-rotation, well above 5° tolerance). + constexpr float32 k_DegToRad = 3.14159265358979323846f / 180.0f; + + auto setupEBSD = [&](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, ChoicesParameter::ValueType neighborScheme) { + const ShapeType cellShape = {3, 3, 3}; + auto& am = ds.getDataRefAs(cellDataPath); + auto& geom = ds.getDataRefAs(geomPath); + + // Quaternions: background = 60° X-rotation, pairs = identity (EBSDlib order: x,y,z,w) + const float32 bgHalf = 60.0f * k_DegToRad * 0.5f; + auto quatsDS = DataStoreUtilities::CreateDataStore(cellShape, {4}, IDataAction::Mode::Execute); + auto* quatsArr = DataArray::Create(ds, "Quats", quatsDS, am.getId()); + auto& quatsStore = quatsArr->getDataStoreRef(); + for(usize i = 0; i < 27; i++) + { + quatsStore[i * 4 + 0] = std::sin(bgHalf); + quatsStore[i * 4 + 1] = 0.0f; + quatsStore[i * 4 + 2] = 0.0f; + quatsStore[i * 4 + 3] = std::cos(bgHalf); + } + // Pair A,B: identity quat at (0,0,0) and (1,1,1) + for(usize idx : {static_cast(0), static_cast(1 * 9 + 1 * 3 + 1)}) + { + quatsStore[idx * 4 + 0] = 0.0f; + quatsStore[idx * 4 + 1] = 0.0f; + quatsStore[idx * 4 + 2] = 0.0f; + quatsStore[idx * 4 + 3] = 1.0f; + } + // Pair C,D: 30° X-rotation at (2,0,0) and (2,1,1) + const float32 pairHalf = 30.0f * k_DegToRad * 0.5f; + for(usize idx : {static_cast(0 * 9 + 0 * 3 + 2), static_cast(1 * 9 + 1 * 3 + 2)}) + { + quatsStore[idx * 4 + 0] = std::sin(pairHalf); + quatsStore[idx * 4 + 1] = 0.0f; + quatsStore[idx * 4 + 2] = 0.0f; + quatsStore[idx * 4 + 3] = std::cos(pairHalf); + } + + // Phases: all phase 1 + auto phasesDS = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* phasesArr = DataArray::Create(ds, "Phases", phasesDS, am.getId()); + phasesArr->fill(1); + + // CrystalStructures: phase 0 = unknown, phase 1 = Cubic_High + const ShapeType ensShape = {2}; + auto* ensAM = AttributeMatrix::Create(ds, "CellEnsembleData", ensShape, geom.getId()); + auto crystDS = DataStoreUtilities::CreateDataStore(ensShape, {1}, IDataAction::Mode::Execute); + auto* crystArr = DataArray::Create(ds, "CrystalStructures", crystDS, ensAM->getId()); + auto& crystStore = crystArr->getDataStoreRef(); + crystStore[0] = 999; + crystStore[1] = 1; // Cubic_High + + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0f)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborScheme)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(DataPath{})); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_IsPeriodic_Key, std::make_any(false)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(geomPath)); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(cellDataPath.createChildPath("Quats"))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(cellDataPath.createChildPath("Phases"))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(DataPath({"Geom", "CellEnsembleData", "CrystalStructures"}))); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any("CellFeatureData")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + }; - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); + RunFaceEdgeVertexConnectivityTest([&](Arguments& args, DataStructure& ds, const DataPath& gp, const DataPath& cp) { setupEBSD(args, ds, gp, cp, 0); }, + [&](Arguments& args, DataStructure& ds, const DataPath& gp, const DataPath& cp) { setupEBSD(args, ds, gp, cp, 1); }); } -TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures:MaskAll", "[OrientationAnalysis][EBSDSegmentFeatures]") +TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: Generate Test Data", "[OrientationAnalysis][EBSDSegmentFeatures][.GenerateTestData]") { UnitTest::LoadPlugins(); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_test_data.tar.gz", "segment_features_test_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/segment_features_test_data/segment_features_test_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + const auto outputDir = fs::path(fmt::format("{}/generated_test_data/ebsd_segment_features", unit_test::k_BinaryTestOutputDir)); + fs::create_directories(outputDir); - // EBSD Segment Features/Semgent Features (Misorientation) Filter + // Small input data (15^3) — one geometry per test variant { - EBSDSegmentFeaturesFilter filter; - Arguments args; - - // Create default Parameters for the filter. - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MisorientationTolerance_Key, std::make_any(5.0F)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(1)); + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_UseMask_Key, std::make_any(true)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_MaskArrayPath)); + DataStructure ds; - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_SelectedImageGeometryPath_Key, std::make_any(ebsd_segment_features_constants::k_InputGeometryPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_QuatsArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_QuatsArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellPhasesArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_PhasesArrayPath)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CrystalStructuresArrayPath_Key, std::make_any(ebsd_segment_features_constants::k_CrystalStructuresArrayPath)); + auto* amBase = BuildSegmentFeaturesTestGeometry(ds, dims, "Base", std::string(k_CellDataName)); + auto& geomBase = ds.getDataRefAs(DataPath({"Base"})); + BuildOrientationTestData(ds, cellShape, geomBase.getId(), amBase->getId(), 1, k_SmallBlockSize); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_FeatureIdsArrayName_Key, std::make_any(k_FeatureIds)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_CellFeatureAttributeMatrixName_Key, std::make_any(k_Grain_Data)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - args.insertOrAssign(EBSDSegmentFeaturesFilter::k_RandomizeFeatureIds_Key, std::make_any(false)); + auto* amMasked = BuildSegmentFeaturesTestGeometry(ds, dims, "Masked", std::string(k_CellDataName)); + auto& geomMasked = ds.getDataRefAs(DataPath({"Masked"})); + BuildOrientationTestData(ds, cellShape, geomMasked.getId(), amMasked->getId(), 1, k_SmallBlockSize); + BuildSphericalMask(ds, cellShape, amMasked->getId()); - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto* amPeriodic = BuildSegmentFeaturesTestGeometry(ds, dims, "Periodic", std::string(k_CellDataName)); + auto& geomPeriodic = ds.getDataRefAs(DataPath({"Periodic"})); + BuildOrientationTestData(ds, cellShape, geomPeriodic.getId(), amPeriodic->getId(), 1, k_SmallBlockSize, true); // wrapBoundary - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + UnitTest::WriteTestDataStructure(ds, outputDir / "small_input.dream3d"); } + // Large input data (200^3) — periodic=true, no mask (sphere mask would eliminate boundary voxels, defeating periodic) { - UInt8Array& actives = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_ActivesArrayPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 32); - } + const ShapeType cellShape = {k_LargeDim, k_LargeDim, k_LargeDim}; + const std::array dims = {k_LargeDim, k_LargeDim, k_LargeDim}; - // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group - { - const auto& generatedDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsArrayPath); - const auto& exemplarDataArray = dataStructure.getDataRefAs(ebsd_segment_features_constants::k_FeatureIdsMaskAllPath); + DataStructure ds; + auto* am = BuildSegmentFeaturesTestGeometry(ds, dims, std::string(k_GeomName), std::string(k_CellDataName)); + auto& geom = ds.getDataRefAs(k_GeomPath); + BuildOrientationTestData(ds, cellShape, geom.getId(), am->getId(), 1, k_LargeBlockSize, true); // wrapBoundary - UnitTest::CompareDataArrays(generatedDataArray, exemplarDataArray); + UnitTest::WriteTestDataStructure(ds, outputDir / "large_input.dream3d"); } - - UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); } diff --git a/src/Plugins/OrientationAnalysis/test/NeighborOrientationCorrelationTest.cpp b/src/Plugins/OrientationAnalysis/test/NeighborOrientationCorrelationTest.cpp index 6ddbce09ea..9de028b8d5 100644 --- a/src/Plugins/OrientationAnalysis/test/NeighborOrientationCorrelationTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/NeighborOrientationCorrelationTest.cpp @@ -3,14 +3,20 @@ #include "OrientationAnalysisTestUtils.hpp" #include "simplnx/Core/Application.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/Dream3dImportParameter.hpp" #include "simplnx/Parameters/GeometrySelectionParameter.hpp" +#include "simplnx/Parameters/MultiArraySelectionParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include +#include #include namespace fs = std::filesystem; @@ -38,6 +44,8 @@ using namespace nx::core::UnitTest; TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN100 Pipeline", "[OrientationAnalysis][NeighborOrientationCorrelationFilter]") { UnitTest::LoadPlugins(); + // 1 Z-slice of quats (largest array): 189*201*4*4 = 607824 bytes + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "neighbor_orientation_correlation.tar.gz", "neighbor_orientation_correlation.dream3d"); @@ -99,6 +107,7 @@ TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN10 // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_CellAttributeMatrix)); auto& cellDataGroup = dataStructure.getDataRefAs(k_CellAttributeMatrix); std::vector selectedCellArrays; @@ -110,6 +119,7 @@ TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN10 for(const auto& cellArrayPath : selectedCellArrays) { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(cellArrayPath)); const auto& generatedDataArray = dataStructure.getDataRefAs(cellArrayPath); DataType type = generatedDataArray.getDataType(); @@ -124,6 +134,7 @@ TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN10 continue; } + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(exemplarDataArrayPath)); auto& exemplarDataArray = exemplarDataStructure.getDataRefAs(exemplarDataArrayPath); DataType exemplarType = exemplarDataArray.getDataType(); @@ -194,3 +205,167 @@ TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Small IN10 UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); } + +namespace +{ +const std::string k_GeomName("Image Geometry"); +const std::string k_CellDataName("Cell Data"); + +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_CellDataPath = k_GeomPath.createChildPath(k_CellDataName); +const DataPath k_CIPath = k_CellDataPath.createChildPath("Confidence Index"); +const DataPath k_QuatsPath = k_CellDataPath.createChildPath("Quats"); +const DataPath k_PhasesPath = k_CellDataPath.createChildPath("Phases"); +const DataPath k_CrystalStructuresPath = k_GeomPath.createChildPath("Ensemble Data").createChildPath("CrystalStructures"); + +void BuildTestData(DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ, usize blockSize) +{ + const ShapeType cellTupleShape = {dimZ, dimY, dimX}; + const usize sliceSize = dimX * dimY; + + auto* imageGeom = ImageGeom::Create(dataStructure, k_GeomName); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(dataStructure, k_CellDataName, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto quatsDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {4}, IDataAction::Mode::Execute); + auto* quatsArray = DataArray::Create(dataStructure, "Quats", quatsDataStore, cellAM->getId()); + auto& quatsStore = quatsArray->getDataStoreRef(); + + auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* phasesArray = DataArray::Create(dataStructure, "Phases", phasesDataStore, cellAM->getId()); + auto& phasesStore = phasesArray->getDataStoreRef(); + + auto ciDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* ciArray = DataArray::Create(dataStructure, "Confidence Index", ciDataStore, cellAM->getId()); + auto& ciStore = ciArray->getDataStoreRef(); + + const usize blocksPerDimX = dimX / blockSize; + const usize blocksPerDimY = dimY / blockSize; + + std::vector quatsBuf(sliceSize * 4); + std::vector phasesBuf(sliceSize); + std::vector ciBuf(sliceSize); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize inSlice = y * dimX + x; + phasesBuf[inSlice] = 1; + + usize bx = x / blockSize; + usize by = y / blockSize; + usize bz = z / blockSize; + float32 angle = static_cast(bz * blocksPerDimY * blocksPerDimX + by * blocksPerDimX + bx) * 0.1f; + float32 sinHalf = std::sin(angle * 0.5f); + float32 cosHalf = std::cos(angle * 0.5f); + + const usize qIdx = inSlice * 4; + quatsBuf[qIdx] = cosHalf; + quatsBuf[qIdx + 1] = sinHalf * 0.577350269f; // 1/sqrt(3) + quatsBuf[qIdx + 2] = sinHalf * 0.577350269f; + quatsBuf[qIdx + 3] = sinHalf * 0.577350269f; + + bool isBoundary = (x % blockSize == 0) || (y % blockSize == 0) || (z % blockSize == 0); + bool isNoisy = ((x * 7 + y * 13 + z * 29) % 10 == 0); + ciBuf[inSlice] = (isBoundary || isNoisy) ? 0.05f : 0.9f; + } + } + const usize zOffset = z * sliceSize; + quatsStore.copyFromBuffer(zOffset * 4, nonstd::span(quatsBuf.data(), sliceSize * 4)); + phasesStore.copyFromBuffer(zOffset, nonstd::span(phasesBuf.data(), sliceSize)); + ciStore.copyFromBuffer(zOffset, nonstd::span(ciBuf.data(), sliceSize)); + } + + // Ensemble data — small enough for per-element writes + auto* ensembleAM = AttributeMatrix::Create(dataStructure, "Ensemble Data", {2}, imageGeom->getId()); + auto crystalStructuresDataStore = DataStoreUtilities::CreateDataStore({2}, {1}, IDataAction::Mode::Execute); + auto* crystalStructuresArray = DataArray::Create(dataStructure, "CrystalStructures", crystalStructuresDataStore, ensembleAM->getId()); + std::array csData = {999, 1}; // Unknown, Cubic-High (m-3m) + crystalStructuresArray->getDataStoreRef().copyFromBuffer(0, nonstd::span(csData.data(), 2)); +} +} // namespace + +TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: Generate Test Data", "[OrientationAnalysis][NeighborOrientationCorrelationFilter][.GenerateTestData]") +{ + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data" / "neighbor_orientation_correlation"; + fs::create_directories(outputDir); + + // Large input data (200x200x200, blockSize=25) + { + DataStructure buildDS; + BuildTestData(buildDS, 200, 200, 200, 25); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "large_input.dream3d"); + fmt::print("Generated large input: {}\n", (outputDir / "large_input.dream3d").string()); + } +} + +TEST_CASE("OrientationAnalysis::NeighborOrientationCorrelationFilter: 200x200x200 Large OOC", "[OrientationAnalysis][NeighborOrientationCorrelationFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 200x200x200, Quats (float32, 4-comp) => 200*200*4*4 = 640,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 640000, true); + + DYNAMIC_SECTION("forceOoc: " << forceOocAlgo) + { + constexpr usize k_Dim = 200; + constexpr usize k_Block = 25; + + DataStructure dataStructure; + BuildTestData(dataStructure, k_Dim, k_Dim, k_Dim, k_Block); + + const NeighborOrientationCorrelationFilter filter; + Arguments args; + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_ImageGeometryPath_Key, std::make_any(k_GeomPath)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_MinConfidence_Key, std::make_any(0.2f)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_MisorientationTolerance_Key, std::make_any(5.0f)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_Level_Key, std::make_any(2)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_CorrelationArrayPath_Key, std::make_any(k_CIPath)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_CellPhasesArrayPath_Key, std::make_any(k_PhasesPath)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_QuatsArrayPath_Key, std::make_any(k_QuatsPath)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_CrystalStructuresArrayPath_Key, std::make_any(k_CrystalStructuresPath)); + args.insertOrAssign(NeighborOrientationCorrelationFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // Some low-CI voxels should have been modified — use Z-slice batched reads for OOC efficiency + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_CIPath)); + const auto& ciAfter = dataStructure.getDataRefAs(k_CIPath).getDataStoreRef(); + const usize sliceSize = k_Dim * k_Dim; + std::vector ciBuf(sliceSize); + usize modifiedCount = 0; + for(usize z = 0; z < k_Dim; z++) + { + ciAfter.copyIntoBuffer(z * sliceSize, nonstd::span(ciBuf.data(), sliceSize)); + for(usize y = 0; y < k_Dim; y++) + { + for(usize x = 0; x < k_Dim; x++) + { + const usize inSlice = y * k_Dim + x; + bool wasBoundary = (x % k_Block == 0) || (y % k_Block == 0) || (z % k_Block == 0); + bool wasNoisy = ((x * 7 + y * 13 + z * 29) % 10 == 0); + if((wasBoundary || wasNoisy) && ciBuf[inSlice] != 0.05f) + { + modifiedCount++; + } + } + } + } + REQUIRE(modifiedCount > 0); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} diff --git a/src/Plugins/OrientationAnalysis/test/ReadH5EbsdTest.cpp b/src/Plugins/OrientationAnalysis/test/ReadH5EbsdTest.cpp index 3ec241dc2a..7dbdb4e7a3 100644 --- a/src/Plugins/OrientationAnalysis/test/ReadH5EbsdTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/ReadH5EbsdTest.cpp @@ -58,13 +58,15 @@ TEST_CASE("OrientationAnalysis::ReadH5Ebsd: Valid filter execution", "[Orientati SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); } - // #ifdef SIMPLNX_WRITE_TEST_OUTPUT +#ifdef SIMPLNX_WRITE_TEST_OUTPUT WriteTestDataStructure(dataStructure, fs::path(fmt::format("{}/read_h5ebsd_test.dream3d", unit_test::k_BinaryTestOutputDir))); - // #endif +#endif // Loop and compare each array from the 'Exemplar Data / CellData' to the 'Data Container / CellData' group { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_CellAttributeMatrix)); auto& cellDataGroup = dataStructure.getDataRefAs(Constants::k_CellAttributeMatrix); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_CellEnsembleAttributeMatrixPath)); auto& cellEnsembleDataGroup = dataStructure.getDataRefAs(Constants::k_CellEnsembleAttributeMatrixPath); std::vector selectedArrays; @@ -84,8 +86,10 @@ TEST_CASE("OrientationAnalysis::ReadH5Ebsd: Valid filter execution", "[Orientati { continue; } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(arrayPath)); const auto& generatedDataArray = dataStructure.getDataRefAs(arrayPath); DataType type = generatedDataArray.getDataType(); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(arrayPath)); auto& exemplarDataArray = exemplarDataStructure.getDataRefAs(arrayPath); DataType exemplarType = exemplarDataArray.getDataType(); diff --git a/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp b/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp index ee2f1243d6..3feea0ec5d 100644 --- a/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/RodriguesConvertorTest.cpp @@ -27,7 +27,7 @@ TEST_CASE("OrientationAnalysis::RodriguesConvertorFilter", "[OrientationAnalysis // Build up a simple Float32Array and place default data into the array Float32Array* quats = UnitTest::CreateTestDataArray(dataStructure, k_InputArrayName, {4ULL}, {3ULL}, {}); - for(size_t i = 0; i < 12; i++) + for(usize i = 0; i < 12; i++) { (*quats)[i] = static_cast(i); } @@ -44,10 +44,10 @@ TEST_CASE("OrientationAnalysis::RodriguesConvertorFilter", "[OrientationAnalysis (*exemplarData)[9] = 0.573462F; (*exemplarData)[10] = 0.655386F; (*exemplarData)[11] = 12.2066F; - (*exemplarData)[12] = 0.517892F; - (*exemplarData)[13] = 0.575435F; - (*exemplarData)[14] = 0.632979F; - (*exemplarData)[15] = 17.37815F; + (*exemplarData)[12] = 0.517893F; + (*exemplarData)[13] = 0.575437F; + (*exemplarData)[14] = 0.632980F; + (*exemplarData)[15] = 17.3781F; { // Instantiate the filter, a DataStructure object and an Arguments Object const RodriguesConvertorFilter filter; @@ -67,6 +67,7 @@ TEST_CASE("OrientationAnalysis::RodriguesConvertorFilter", "[OrientationAnalysis auto executeResult = filter.execute(dataStructure, args); SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + REQUIRE_NOTHROW(dataStructure.getDataRefAs(DataPath({k_ConvertedName}))); auto& outputArray = dataStructure.getDataRefAs(DataPath({k_ConvertedName})); UnitTest::CompareDataArrays(*exemplarData, outputArray); diff --git a/src/Plugins/OrientationAnalysis/test/WritePoleFigureTest.cpp b/src/Plugins/OrientationAnalysis/test/WritePoleFigureTest.cpp index d0fd8196f8..d5d7bf165d 100644 --- a/src/Plugins/OrientationAnalysis/test/WritePoleFigureTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/WritePoleFigureTest.cpp @@ -27,9 +27,7 @@ const std::string k_ImagePrefix("fw-ar-IF1-aptr12-corr Discrete Pole Figure"); template void CompareComponentsOfArrays(const DataStructure& dataStructure, const DataPath& exemplaryDataPath, const DataPath& computedPath, usize compIndex) { - // DataPath exemplaryDataPath = featureGroup.createChildPath("SurfaceFeatures"); REQUIRE_NOTHROW(dataStructure.getDataRefAs>(exemplaryDataPath)); - auto* computedData = dataStructure.getData(computedPath); REQUIRE_NOTHROW(dataStructure.getDataRefAs>(computedPath)); const auto& exemplaryDataArray = dataStructure.getDataRefAs>(exemplaryDataPath); @@ -47,12 +45,19 @@ void CompareComponentsOfArrays(const DataStructure& dataStructure, const DataPat INFO(fmt::format("Bad Comparison\n Input Data Array:'{}'\n Output DataArray: '{}'", exemplaryDataPath.toString(), computedPath.toString())); - usize start = 0; - usize numTuples = exemplaryDataArray.getNumberOfTuples(); - for(usize i = start; i < numTuples; i++) + const usize numTuples = exemplaryDataArray.getNumberOfTuples(); + const usize totalElements = numTuples * exemplaryNumComp; + + // Bulk-read both arrays into local buffers to avoid per-element OOC overhead + std::vector exemplarBuf(totalElements); + std::vector generatedBuf(totalElements); + exemplaryDataArray.getDataStoreRef().copyIntoBuffer(0, nonstd::span(exemplarBuf.data(), totalElements)); + generatedDataArray.getDataStoreRef().copyIntoBuffer(0, nonstd::span(generatedBuf.data(), totalElements)); + + for(usize i = 0; i < numTuples; i++) { - auto oldVal = exemplaryDataArray[i * exemplaryNumComp + compIndex]; - auto newVal = generatedDataArray[i * generatedNumComp + compIndex]; + auto oldVal = exemplarBuf[i * exemplaryNumComp + compIndex]; + auto newVal = generatedBuf[i * generatedNumComp + compIndex]; INFO(fmt::format("Index: {} Comp: {}", i, compIndex)); REQUIRE(oldVal == newVal); diff --git a/src/Plugins/SimplnxCore/CMakeLists.txt b/src/Plugins/SimplnxCore/CMakeLists.txt index 30d100f871..e64e9e3e73 100644 --- a/src/Plugins/SimplnxCore/CMakeLists.txt +++ b/src/Plugins/SimplnxCore/CMakeLists.txt @@ -185,7 +185,9 @@ set(AlgorithmList ComputeArrayHistogramByFeature ComputeArrayStatistics ComputeBiasedFeatures + ComputeBoundaryCells ComputeBoundaryCellsDirect + ComputeBoundaryCellsScanline ComputeBoundingBoxStats ComputeCoordinatesImageGeom ComputeCoordinateThreshold @@ -195,21 +197,29 @@ set(AlgorithmList ComputeFeatureBounds ComputeFeatureCentroids ComputeFeatureClustering + ComputeFeatureNeighbors ComputeFeatureNeighborsDirect + ComputeFeatureNeighborsScanline ComputeFeaturePhases ComputeFeaturePhasesBinary ComputeFeatureRect ComputeFeatureSizes ComputeGroupingDensity ComputeKMeans + ComputeKMedoids ComputeKMedoidsDirect + ComputeKMedoidsScanline ComputeLargestCrossSections ComputeMomentInvariants2D ComputeNeighborhoods ComputeNeighborListStatistics # ComputeNumFeatures + ComputeSurfaceAreaToVolume ComputeSurfaceAreaToVolumeDirect + ComputeSurfaceAreaToVolumeScanline + ComputeSurfaceFeatures ComputeSurfaceFeaturesDirect + ComputeSurfaceFeaturesScanline # ComputeTriangleAreas ComputeTriangleGeomCentroids ComputeTriangleGeomVolumes @@ -235,7 +245,9 @@ set(AlgorithmList CropEdgeGeometry CropImageGeometry CropVertexGeometry + DBSCAN DBSCANDirect + DBSCANScanline # DeleteData ErodeDilateBadData ErodeDilateCoordinationNumber @@ -246,12 +258,16 @@ set(AlgorithmList ExtractPipelineToFile ExtractVertexGeometry FeatureFaceCurvature + FillBadData FillBadDataBFS + FillBadDataCCL FindNRingNeighbors FlyingEdges3D HierarchicalSmooth IdentifyDuplicateVertices + IdentifySample IdentifySampleBFS + IdentifySampleCCL InitializeData InitializeImageGeomCellData InterpolatePointCloudToRegularGrid @@ -261,14 +277,18 @@ set(AlgorithmList LaplacianSmoothing MapPointCloudToRegularGrid # MoveData + MultiThresholdObjects MultiThresholdObjectsDirect + MultiThresholdObjectsScanline NearestPointFuseRegularGrids PadImageGeometry PartitionGeometry PointSampleEdgeGeometry ExtractFeatureBoundaries2D PointSampleTriangleGeometry + QuickSurfaceMesh QuickSurfaceMeshDirect + QuickSurfaceMeshScanline ReadBinaryCTNorthstar ReadCSVFile ReadDeformKeyFileV12 @@ -303,7 +323,9 @@ set(AlgorithmList SliceTriangleGeometry SplitDataArrayByComponent SplitDataArrayByTuple + SurfaceNets SurfaceNetsDirect + SurfaceNetsScanline TriangleCentroid TriangleDihedralAngle TriangleNormal diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp index e6adcc7ba8..f3f31ffe90 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp @@ -3,11 +3,10 @@ #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/MaskCompareUtilities.hpp" -#include - using namespace nx::core; // ----------------------------------------------------------------------------- @@ -37,8 +36,16 @@ Result<> AlignSectionsFeatureCentroid::operator()() } // ----------------------------------------------------------------------------- -Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, std::vector& yShifts) +Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, std::vector& yShifts) { + { + const auto& maskCheck = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + if(ForceOocAlgorithm() || IsOutOfCore(maskCheck)) + { + return findShiftsOoc(xShifts, yShifts); + } + } + std::unique_ptr maskCompare; try { @@ -55,23 +62,23 @@ Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, SizeVec3 dims = gridGeom->getDimensions(); - int64_t sdims[3] = { - static_cast(dims[0]), - static_cast(dims[1]), - static_cast(dims[2]), + int64 sdims[3] = { + static_cast(dims[0]), + static_cast(dims[1]), + static_cast(dims[2]), }; - int32_t progInt = 0; + int32 progInt = 0; - size_t slice = 0; - size_t point = 0; + usize slice = 0; + usize point = 0; nx::core::FloatVec3 spacing = gridGeom->getSpacing(); - std::vector xCentroid(dims[2], 0.0f); - std::vector yCentroid(dims[2], 0.0f); + std::vector xCentroid(dims[2], 0.0f); + std::vector yCentroid(dims[2], 0.0f); ThrottledMessenger throttledMessenger = getMessageHelper().createThrottledMessenger(); // Loop over the Z Direction - for(size_t iter = 0; iter < dims[2]; iter++) + for(usize iter = 0; iter < dims[2]; iter++) { if(m_ShouldCancel) { @@ -79,35 +86,35 @@ Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, } throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Determining Shifts || {:.2f}% Complete", CalculatePercentComplete(iter, dims[2])); }); - size_t count = 0; + usize count = 0; xCentroid[iter] = 0; yCentroid[iter] = 0; - slice = static_cast((dims[2] - 1) - iter); - for(size_t l = 0; l < dims[1]; l++) + slice = static_cast((dims[2] - 1) - iter); + for(usize l = 0; l < dims[1]; l++) { - for(size_t n = 0; n < dims[0]; n++) + for(usize n = 0; n < dims[0]; n++) { point = ((slice)*dims[0] * dims[1]) + (l * dims[0]) + n; if(maskCompare->isTrue(point)) { - xCentroid[iter] = xCentroid[iter] + (static_cast(n) * spacing[0]); - yCentroid[iter] = yCentroid[iter] + (static_cast(l) * spacing[1]); + xCentroid[iter] = xCentroid[iter] + (static_cast(n) * spacing[0]); + yCentroid[iter] = yCentroid[iter] + (static_cast(l) * spacing[1]); count++; } } } - xCentroid[iter] = xCentroid[iter] / static_cast(count); - yCentroid[iter] = yCentroid[iter] / static_cast(count); + xCentroid[iter] = xCentroid[iter] / static_cast(count); + yCentroid[iter] = yCentroid[iter] / static_cast(count); } bool xWarning = false; bool yWarning = false; if(m_InputValues->StoreAlignmentShifts) { - size_t relativexshift = 0; - size_t relativeyshift = 0; + usize relativexshift = 0; + usize relativeyshift = 0; auto& slicesStore = m_DataStructure.getDataAs(m_InputValues->SlicesArrayPath)->getDataStoreRef(); auto& relativeShiftsStore = m_DataStructure.getDataAs(m_InputValues->RelativeShiftsArrayPath)->getDataStoreRef(); @@ -115,14 +122,14 @@ Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, auto& centroidsStore = m_DataStructure.getDataAs(m_InputValues->CentroidsArrayPath)->getDataStoreRef(); // Calculate the X&Y shifts based on the centroid. Note the shifts are in real units - for(size_t iter = 1; iter < dims[2]; iter++) + for(usize iter = 1; iter < dims[2]; iter++) { slice = (dims[2] - 1) - iter; if(m_InputValues->UseReferenceSlice) { // Cumulative and Relative are identical - relativexshift = static_cast((xCentroid[iter] - xCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[0]); - relativeyshift = static_cast((yCentroid[iter] - yCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[1]); + relativexshift = static_cast((xCentroid[iter] - xCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[0]); + relativeyshift = static_cast((yCentroid[iter] - yCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[1]); xShifts[iter] = relativexshift; yShifts[iter] = relativeyshift; } @@ -179,17 +186,17 @@ Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, else { // Calculate the X&Y shifts based on the centroid. Note the shifts are in real units - for(size_t iter = 1; iter < dims[2]; iter++) + for(usize iter = 1; iter < dims[2]; iter++) { if(m_InputValues->UseReferenceSlice) { - xShifts[iter] = static_cast((xCentroid[iter] - xCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[0]); - yShifts[iter] = static_cast((yCentroid[iter] - yCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[1]); + xShifts[iter] = static_cast((xCentroid[iter] - xCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[0]); + yShifts[iter] = static_cast((yCentroid[iter] - yCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[1]); } else { - xShifts[iter] = xShifts[iter - 1] + static_cast((xCentroid[iter] - xCentroid[iter - 1]) / spacing[0]); - yShifts[iter] = yShifts[iter - 1] + static_cast((yCentroid[iter] - yCentroid[iter - 1]) / spacing[1]); + xShifts[iter] = xShifts[iter - 1] + static_cast((xCentroid[iter] - xCentroid[iter - 1]) / spacing[0]); + yShifts[iter] = yShifts[iter - 1] + static_cast((yCentroid[iter] - yCentroid[iter - 1]) / spacing[1]); } if((xShifts[iter] < -sdims[0] || xShifts[iter] > sdims[0]) && !xWarning) @@ -225,3 +232,171 @@ Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, return {}; } + +// ----------------------------------------------------------------------------- +// OOC-optimized findShifts: bulk-reads mask per Z-slice instead of per-element +// isTrue() calls, eliminating OOC chunk thrashing in the centroid computation. +// ----------------------------------------------------------------------------- +Result<> AlignSectionsFeatureCentroid::findShiftsOoc(std::vector& xShifts, std::vector& yShifts) +{ + // Get raw mask store for bulk reads + const auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + const AbstractDataStore* maskUInt8StorePtr = nullptr; + const AbstractDataStore* maskBoolStorePtr = nullptr; + if(maskArray.getDataType() == DataType::uint8) + { + maskUInt8StorePtr = &dynamic_cast&>(maskArray).getDataStoreRef(); + } + else if(maskArray.getDataType() == DataType::boolean) + { + maskBoolStorePtr = &dynamic_cast&>(maskArray).getDataStoreRef(); + } + else + { + return MakeErrorResult(-53900, fmt::format("Mask Array is not Bool or UInt8: {}", m_InputValues->MaskArrayPath.toString())); + } + + auto* gridGeom = m_DataStructure.getDataAs(m_InputValues->ImageGeometryPath); + SizeVec3 dims = gridGeom->getDimensions(); + + int64 sdims[3] = { + static_cast(dims[0]), + static_cast(dims[1]), + static_cast(dims[2]), + }; + + nx::core::FloatVec3 spacing = gridGeom->getSpacing(); + std::vector xCentroid(dims[2], 0.0f); + std::vector yCentroid(dims[2], 0.0f); + + const usize sliceVoxels = dims[0] * dims[1]; + std::vector maskBuf(sliceVoxels); + + // Compute centroids per Z-slice using bulk mask reads + for(usize iter = 0; iter < dims[2]; iter++) + { + if(m_ShouldCancel) + { + return {}; + } + + usize slice = static_cast((dims[2] - 1) - iter); + usize sliceOffset = slice * sliceVoxels; + + // Bulk-read mask for this slice + if(maskUInt8StorePtr != nullptr) + { + maskUInt8StorePtr->copyIntoBuffer(sliceOffset, nonstd::span(maskBuf.data(), sliceVoxels)); + } + else if(maskBoolStorePtr != nullptr) + { + auto boolBuf = std::make_unique(sliceVoxels); + maskBoolStorePtr->copyIntoBuffer(sliceOffset, nonstd::span(boolBuf.get(), sliceVoxels)); + for(usize idx = 0; idx < sliceVoxels; idx++) + { + maskBuf[idx] = boolBuf[idx] ? 1 : 0; + } + } + + usize count = 0; + xCentroid[iter] = 0; + yCentroid[iter] = 0; + + for(usize l = 0; l < dims[1]; l++) + { + for(usize n = 0; n < dims[0]; n++) + { + usize localIdx = l * dims[0] + n; + if(maskBuf[localIdx] != 0) + { + xCentroid[iter] += static_cast(n) * spacing[0]; + yCentroid[iter] += static_cast(l) * spacing[1]; + count++; + } + } + } + xCentroid[iter] = xCentroid[iter] / static_cast(count); + yCentroid[iter] = yCentroid[iter] / static_cast(count); + } + + // Calculate shifts from centroids (same logic as in-core path) + bool xWarning = false; + bool yWarning = false; + for(usize iter = 1; iter < dims[2]; iter++) + { + if(m_InputValues->UseReferenceSlice) + { + xShifts[iter] = static_cast((xCentroid[iter] - xCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[0]); + yShifts[iter] = static_cast((yCentroid[iter] - yCentroid[static_cast(m_InputValues->ReferenceSlice)]) / spacing[1]); + } + else + { + xShifts[iter] = xShifts[iter - 1] + static_cast((xCentroid[iter] - xCentroid[iter - 1]) / spacing[0]); + yShifts[iter] = yShifts[iter - 1] + static_cast((yCentroid[iter] - yCentroid[iter - 1]) / spacing[1]); + } + + if((xShifts[iter] < -sdims[0] || xShifts[iter] > sdims[0]) && !xWarning) + { + m_MessageHandler(nx::core::IFilter::Message::Type::Info, fmt::format("A shift was greater than the X dimension of the Image Geometry. " + "All subsequent slices are probably wrong. Slice={} X Dim={} X Shift={} sDims[0]={}", + iter, dims[0], xShifts[iter], sdims[0])); + xWarning = true; + } + if((yShifts[iter] < -sdims[1] || yShifts[iter] > sdims[1]) && !yWarning) + { + m_MessageHandler(nx::core::IFilter::Message::Type::Info, fmt::format("A shift was greater than the Y dimension of the Image Geometry. " + "All subsequent slices are probably wrong. Slice={} Y Dim={} Y Shift={} sDims[1]={}", + iter, dims[1], yShifts[iter], sdims[1])); + yWarning = true; + } + if(std::isnan(xCentroid[iter]) && !xWarning) + { + m_MessageHandler(nx::core::IFilter::Message::Type::Info, fmt::format("The X Centroid was NaN. All subsequent slices are probably wrong. Slice=", iter)); + xWarning = true; + } + if(std::isnan(yCentroid[iter]) && !yWarning) + { + m_MessageHandler(nx::core::IFilter::Message::Type::Info, fmt::format("The Y Centroid was NaN. All subsequent slices are probably wrong. Slice=", iter)); + yWarning = true; + } + } + + // Store alignment shifts if requested + if(m_InputValues->StoreAlignmentShifts) + { + auto& slicesStore = m_DataStructure.getDataAs(m_InputValues->SlicesArrayPath)->getDataStoreRef(); + auto& relativeShiftsStore = m_DataStructure.getDataAs(m_InputValues->RelativeShiftsArrayPath)->getDataStoreRef(); + auto& cumulativeShiftsStore = m_DataStructure.getDataAs(m_InputValues->CumulativeShiftsArrayPath)->getDataStoreRef(); + auto& centroidsStore = m_DataStructure.getDataAs(m_InputValues->CentroidsArrayPath)->getDataStoreRef(); + + for(usize iter = 1; iter < dims[2]; iter++) + { + usize slice = (dims[2] - 1) - iter; + int64 relativexshift = 0; + int64 relativeyshift = 0; + if(m_InputValues->UseReferenceSlice) + { + relativexshift = xShifts[iter]; + relativeyshift = yShifts[iter]; + } + else + { + relativexshift = static_cast((xCentroid[iter] - xCentroid[iter - 1]) / spacing[0]); + relativeyshift = static_cast((yCentroid[iter] - yCentroid[iter - 1]) / spacing[1]); + } + + usize xIndex = iter * 2; + usize yIndex = (iter * 2) + 1; + slicesStore[xIndex] = slice; + slicesStore[yIndex] = slice + 1; + relativeShiftsStore[xIndex] = relativexshift; + relativeShiftsStore[yIndex] = relativeyshift; + cumulativeShiftsStore[xIndex] = xShifts[iter]; + cumulativeShiftsStore[yIndex] = yShifts[iter]; + centroidsStore[xIndex] = xCentroid[iter]; + centroidsStore[yIndex] = yCentroid[iter]; + } + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp index c670c82741..134d054bfa 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp @@ -50,9 +50,11 @@ class SIMPLNXCORE_EXPORT AlignSectionsFeatureCentroid : public AlignSections * @param yShifts * @return Whether the x and y shifts were successfully found */ - Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; + Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; private: + Result<> findShiftsOoc(std::vector& xShifts, std::vector& yShifts); + DataStructure& m_DataStructure; const AlignSectionsFeatureCentroidInputValues* m_InputValues = nullptr; const std::atomic_bool& m_ShouldCancel; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AppendImageGeometry.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AppendImageGeometry.cpp index a3b344da03..039b29f944 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AppendImageGeometry.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AppendImageGeometry.cpp @@ -64,11 +64,10 @@ Result<> AppendImageGeometry::operator()() ParallelTaskAlgorithm taskRunner; for(const auto& [dataId, dataObject] : *newCellData) { - if(getCancel()) + if(m_ShouldCancel) { return {}; } - const std::string name = dataObject->getName(); auto newDataArrayPath = newCellDataPath.createChildPath(name); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp index 84861d6480..a2c0a168db 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp @@ -207,7 +207,7 @@ class StatisticsByFeatureImpl { throttledMessenger.sendThrottledMessage([&]() { progressCount = 0; - return fmt::format("StdDev Calculation Feature/Ensemble [{}-{}]: {:.2f}%", start, end, 100.0f * static_cast(tupleIndex) / static_cast(numTuples)); + return fmt::format("StdDev Calculation Feature/Ensemble [{}-{}]: {:.2f}%", start, end, 100.0f * static_cast(tupleIndex) / static_cast(numTuples)); }); } } @@ -543,7 +543,7 @@ class MedianByFeatureRangeImpl { if(m_FindMedian) { - values.push_back(static_cast(m_Source[i])); + values.push_back(static_cast(m_Source[i])); } if(m_FindNumUniqueValues) { @@ -640,7 +640,7 @@ Result<> FindStatisticsImpl(const ContainerType& data, std::vector& arr // Finding the mean depends on the summation. if(inputValues->FindSummation || inputValues->FindMean || inputValues->FindStdDeviation) { - const std::pair sumMeanValues = StatisticsCalculations::FindSumMean(data); + const std::pair sumMeanValues = StatisticsCalculations::FindSumMean(data); if(inputValues->FindSummation) { auto* array6Ptr = dynamic_cast(arrays[7]); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp new file mode 100644 index 0000000000..1f6ad3a5c3 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp @@ -0,0 +1,34 @@ +#include "ComputeBoundaryCells.hpp" + +#include "ComputeBoundaryCellsDirect.hpp" +#include "ComputeBoundaryCellsScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeBoundaryCells::ComputeBoundaryCells(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeBoundaryCells::~ComputeBoundaryCells() noexcept = default; + +// ----------------------------------------------------------------------------- +const std::atomic_bool& ComputeBoundaryCells::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> ComputeBoundaryCells::operator()() +{ + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp new file mode 100644 index 0000000000..954173dde4 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp @@ -0,0 +1,46 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT ComputeBoundaryCellsInputValues +{ + bool IgnoreFeatureZero; + bool IncludeVolumeBoundary; + DataPath ImageGeometryPath; + DataPath FeatureIdsArrayPath; + DataPath BoundaryCellsArrayName; +}; + +/** + * @class + */ +class SIMPLNXCORE_EXPORT ComputeBoundaryCells +{ +public: + ComputeBoundaryCells(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues); + ~ComputeBoundaryCells() noexcept; + + ComputeBoundaryCells(const ComputeBoundaryCells&) = delete; + ComputeBoundaryCells(ComputeBoundaryCells&&) noexcept = delete; + ComputeBoundaryCells& operator=(const ComputeBoundaryCells&) = delete; + ComputeBoundaryCells& operator=(ComputeBoundaryCells&&) noexcept = delete; + + Result<> operator()(); + + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp index 91e50d6aba..9bbcea5def 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp @@ -1,5 +1,7 @@ #include "ComputeBoundaryCellsDirect.hpp" +#include "ComputeBoundaryCells.hpp" + #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" @@ -7,7 +9,8 @@ using namespace nx::core; // ----------------------------------------------------------------------------- -ComputeBoundaryCellsDirect::ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues) +ComputeBoundaryCellsDirect::ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeBoundaryCellsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -19,15 +22,13 @@ ComputeBoundaryCellsDirect::ComputeBoundaryCellsDirect(DataStructure& dataStruct ComputeBoundaryCellsDirect::~ComputeBoundaryCellsDirect() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ComputeBoundaryCellsDirect::getCancel() -{ - return m_ShouldCancel; -} - -// ----------------------------------------------------------------------------- +/** + * @brief Counts boundary faces per voxel using direct Z-Y-X iteration. + * In-core path: iterates all voxels sequentially, checking 6 face neighbors. + */ Result<> ComputeBoundaryCellsDirect::operator()() { - const ImageGeom imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); + const auto& imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const SizeVec3 udims = imageGeometry.getDimensions(); std::array dims = { static_cast(udims[0]), diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp index a2e833245b..b2ed65ffe4 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp @@ -2,29 +2,23 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" namespace nx::core { - -struct SIMPLNXCORE_EXPORT ComputeBoundaryCellsInputValues -{ - bool IgnoreFeatureZero; - bool IncludeVolumeBoundary; - DataPath ImageGeometryPath; - DataPath FeatureIdsArrayPath; - DataPath BoundaryCellsArrayName; -}; +struct ComputeBoundaryCellsInputValues; /** - * @class + * @class ComputeBoundaryCellsDirect + * @brief In-core algorithm for ComputeBoundaryCells. Preserves the original sequential + * Z-Y-X voxel iteration with face-neighbor boundary counting. Selected by DispatchAlgorithm + * when all input arrays are backed by in-memory DataStore. */ class SIMPLNXCORE_EXPORT ComputeBoundaryCellsDirect { public: - ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues); + ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeBoundaryCellsInputValues* inputValues); ~ComputeBoundaryCellsDirect() noexcept; ComputeBoundaryCellsDirect(const ComputeBoundaryCellsDirect&) = delete; @@ -34,8 +28,6 @@ class SIMPLNXCORE_EXPORT ComputeBoundaryCellsDirect Result<> operator()(); - const std::atomic_bool& getCancel(); - private: DataStructure& m_DataStructure; const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp new file mode 100644 index 0000000000..3176e89a2a --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp @@ -0,0 +1,177 @@ +#include "ComputeBoundaryCellsScanline.hpp" + +#include "ComputeBoundaryCells.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include +#include + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeBoundaryCellsScanline::ComputeBoundaryCellsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeBoundaryCellsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeBoundaryCellsScanline::~ComputeBoundaryCellsScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief Counts boundary faces per voxel using Z-slice rolling window iteration. + * OOC path: reads/writes one Z-slice at a time via copyIntoBuffer/copyFromBuffer, + * keeping a 3-slice rolling window (prevSlice, curSlice, nextSlice) for Z-neighbor access. + */ +Result<> ComputeBoundaryCellsScanline::operator()() +{ + const auto& imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); + const SizeVec3 udims = imageGeometry.getDimensions(); + const int64 dimX = static_cast(udims[0]); + const int64 dimY = static_cast(udims[1]); + const int64 dimZ = static_cast(udims[2]); + + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + auto& boundaryCellsStore = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsArrayName)->getDataStoreRef(); + + int32 ignoreFeatureZeroVal = 0; + if(!m_InputValues->IgnoreFeatureZero) + { + ignoreFeatureZeroVal = -1; + } + + const usize sliceSize = static_cast(dimY) * static_cast(dimX); + + std::vector prevSlice(sliceSize); + std::vector curSlice(sliceSize); + std::vector nextSlice(sliceSize); + std::vector outputSlice(sliceSize); + + // Load first Z-slice + featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + if(dimZ > 1) + { + featureIdsStore.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + + for(int64 zIdx = 0; zIdx < dimZ; zIdx++) + { + if(m_ShouldCancel) + { + return {}; + } + // Process current slice + for(int64 yIdx = 0; yIdx < dimY; yIdx++) + { + const int64 rowOffset = yIdx * dimX; + for(int64 xIdx = 0; xIdx < dimX; xIdx++) + { + const int64 sliceIndex = rowOffset + xIdx; + int8 onSurf = 0; + const int32 feature = curSlice[sliceIndex]; + + if(feature >= 0) + { + if(m_InputValues->IncludeVolumeBoundary) + { + if(dimX > 2 && (xIdx == 0 || xIdx == dimX - 1)) + { + onSurf++; + } + if(dimY > 2 && (yIdx == 0 || yIdx == dimY - 1)) + { + onSurf++; + } + if(dimZ > 2 && (zIdx == 0 || zIdx == dimZ - 1)) + { + onSurf++; + } + + if(onSurf > 0 && feature == 0) + { + onSurf = 0; + } + } + + // -Z neighbor + if(zIdx > 0) + { + if(prevSlice[sliceIndex] != feature && prevSlice[sliceIndex] > ignoreFeatureZeroVal) + { + onSurf++; + } + } + + // -Y neighbor + if(yIdx > 0) + { + const int64 neighborIdx = sliceIndex - dimX; + if(curSlice[neighborIdx] != feature && curSlice[neighborIdx] > ignoreFeatureZeroVal) + { + onSurf++; + } + } + + // -X neighbor + if(xIdx > 0) + { + const int64 neighborIdx = sliceIndex - 1; + if(curSlice[neighborIdx] != feature && curSlice[neighborIdx] > ignoreFeatureZeroVal) + { + onSurf++; + } + } + + // +X neighbor + if(xIdx < dimX - 1) + { + const int64 neighborIdx = sliceIndex + 1; + if(curSlice[neighborIdx] != feature && curSlice[neighborIdx] > ignoreFeatureZeroVal) + { + onSurf++; + } + } + + // +Y neighbor + if(yIdx < dimY - 1) + { + const int64 neighborIdx = sliceIndex + dimX; + if(curSlice[neighborIdx] != feature && curSlice[neighborIdx] > ignoreFeatureZeroVal) + { + onSurf++; + } + } + + // +Z neighbor + if(zIdx < dimZ - 1) + { + if(nextSlice[sliceIndex] != feature && nextSlice[sliceIndex] > ignoreFeatureZeroVal) + { + onSurf++; + } + } + } + + outputSlice[sliceIndex] = onSurf; + } + } + + // Write output slice + boundaryCellsStore.copyFromBuffer(static_cast(zIdx) * sliceSize, nonstd::span(outputSlice.data(), sliceSize)); + + // Shift rolling window + std::swap(prevSlice, curSlice); + std::swap(curSlice, nextSlice); + if(zIdx + 2 < dimZ) + { + featureIdsStore.copyIntoBuffer(static_cast(zIdx + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp new file mode 100644 index 0000000000..0628f62a5f --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp @@ -0,0 +1,39 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct ComputeBoundaryCellsInputValues; + +/** + * @class ComputeBoundaryCellsScanline + * @brief Out-of-core algorithm for ComputeBoundaryCells. Iterates Z-slices sequentially + * using copyIntoBuffer/copyFromBuffer bulk I/O with a 3-slice rolling window + * (prevSlice, curSlice, nextSlice) for Z-neighbor access. Selected by DispatchAlgorithm + * when any input array is backed by ZarrStore. + */ +class SIMPLNXCORE_EXPORT ComputeBoundaryCellsScanline +{ +public: + ComputeBoundaryCellsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeBoundaryCellsInputValues* inputValues); + ~ComputeBoundaryCellsScanline() noexcept; + + ComputeBoundaryCellsScanline(const ComputeBoundaryCellsScanline&) = delete; + ComputeBoundaryCellsScanline(ComputeBoundaryCellsScanline&&) noexcept = delete; + ComputeBoundaryCellsScanline& operator=(const ComputeBoundaryCellsScanline&) = delete; + ComputeBoundaryCellsScanline& operator=(ComputeBoundaryCellsScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp index 1469bd95a9..ebc29363ff 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp @@ -5,26 +5,42 @@ #include "simplnx/Utilities/NeighborUtilities.hpp" #include "simplnx/Utilities/ParallelTaskAlgorithm.hpp" +#include + using namespace nx::core; namespace { /** - * @brief The ComputeDistanceMapImpl class implements a threaded algorithm that computes the distance map - * for each point in the supplied volume + * @brief The ComputeDistanceMapImpl class implements a threaded algorithm that computes the distance map + * for each point in the supplied volume. + * + * Accepts pre-buffered local arrays instead of DataStore references to avoid + * per-element virtual dispatch overhead (important for OOC stores). */ template class ComputeDistanceMapImpl { - DataStructure& m_DataStructure; const ComputeEuclideanDistMapInputValues& m_InputValues; std::vector& m_NearestNeighbors; + const int32* m_FeatureIds = nullptr; + T* m_DistBuf = nullptr; + AbstractDataStore* m_OutputStore = nullptr; + usize m_TotalVoxels = 0; + SizeVec3 m_Dims = {}; + FloatVec3 m_Spacing = {}; public: - ComputeDistanceMapImpl(DataStructure& dataStructure, const ComputeEuclideanDistMapInputValues& inputValues, std::vector& nearestNeighbors) - : m_DataStructure(dataStructure) - , m_InputValues(inputValues) + ComputeDistanceMapImpl(const ComputeEuclideanDistMapInputValues& inputValues, std::vector& nearestNeighbors, const int32* featureIds, T* distBuf, AbstractDataStore* outputStore, + usize totalVoxels, SizeVec3 dims, FloatVec3 spacing) + : m_InputValues(inputValues) , m_NearestNeighbors(nearestNeighbors) + , m_FeatureIds(featureIds) + , m_DistBuf(distBuf) + , m_OutputStore(outputStore) + , m_TotalVoxels(totalVoxels) + , m_Dims(dims) + , m_Spacing(spacing) { } @@ -32,39 +48,15 @@ class ComputeDistanceMapImpl void operator()() const { - using DataArrayType = DataArray; - using DataStoreType = AbstractDataStore; - - const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues.InputImageGeometry); + auto xpoints = static_cast(m_Dims[0]); + auto ypoints = static_cast(m_Dims[1]); + auto zpoints = static_cast(m_Dims[2]); - DataStoreType* gbManhattanDistancesStore = nullptr; - if(m_InputValues.DoBoundaries) - { - gbManhattanDistancesStore = m_DataStructure.template getDataAs(m_InputValues.GBDistancesArrayPath)->getDataStore(); - } - DataStoreType* tjManhattanDistancesStore = nullptr; - if(m_InputValues.DoTripleLines) - { - tjManhattanDistancesStore = m_DataStructure.template getDataAs(m_InputValues.TJDistancesArrayPath)->getDataStore(); - } - DataStoreType* qpManhattanDistancesStore = nullptr; - if(m_InputValues.DoQuadPoints) - { - qpManhattanDistancesStore = m_DataStructure.template getDataAs(m_InputValues.QPDistancesArrayPath)->getDataStore(); - } - - SizeVec3 udims = selectedImageGeom.getDimensions(); - - size_t totalVoxels = selectedImageGeom.getNumberOfCells(); float64 Distance = 0.0; - size_t count = 1; - size_t changed = 1; - size_t neighpoint = 0; + usize count = 1; + usize changed = 1; + usize neighpoint = 0; int64_t neighbors[6] = {0, 0, 0, 0, 0, 0}; - auto xpoints = static_cast(udims[0]); - auto ypoints = static_cast(udims[1]); - auto zpoints = static_cast(udims[2]); - FloatVec3 spacing = selectedImageGeom.getSpacing(); neighbors[0] = -xpoints * ypoints; neighbors[1] = -xpoints; @@ -73,17 +65,13 @@ class ComputeDistanceMapImpl neighbors[4] = xpoints; neighbors[5] = xpoints * ypoints; - std::vector voxel_NearestNeighbor(totalVoxels, 0); - std::vector voxel_Distance(totalVoxels, 0.0); - - // Input Arrays - const auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues.FeatureIdsArrayPath)->getDataStoreRef(); + std::vector voxel_NearestNeighbor(m_TotalVoxels, 0); + std::vector voxel_Distance(m_TotalVoxels, 0.0); Distance = 0; // This loop initializes the `voxel_NearestNeighbor` and `voxel_Distance` temp arrays with values - for(size_t voxelTupleIdx = 0; voxelTupleIdx < totalVoxels; ++voxelTupleIdx) + for(usize voxelTupleIdx = 0; voxelTupleIdx < m_TotalVoxels; ++voxelTupleIdx) { - // For the given `mapType`, get the value that was stored for the nearestNeighbor, // essentially, as long as the value is **NOT** -1. if(m_NearestNeighbors[voxelTupleIdx * 3 + static_cast(MapType)] >= 0) @@ -96,18 +84,7 @@ class ComputeDistanceMapImpl // If a default value was stored into the NearestNeighbor then set that into the voxel_NearestNeighbor vector at the current voxel index voxel_NearestNeighbor[voxelTupleIdx] = -1; } - if(m_InputValues.DoBoundaries && MapType == ComputeEuclideanDistMap::MapType::FeatureBoundary) - { - voxel_Distance[voxelTupleIdx] = static_cast((*gbManhattanDistancesStore)[voxelTupleIdx]); - } - else if(m_InputValues.DoTripleLines && MapType == ComputeEuclideanDistMap::MapType::TripleJunction) - { - voxel_Distance[voxelTupleIdx] = static_cast((*tjManhattanDistancesStore)[voxelTupleIdx]); - } - else if(m_InputValues.DoQuadPoints && MapType == ComputeEuclideanDistMap::MapType::QuadPoint) - { - voxel_Distance[voxelTupleIdx] = static_cast((*qpManhattanDistancesStore)[voxelTupleIdx]); - } + voxel_Distance[voxelTupleIdx] = static_cast(m_DistBuf[voxelTupleIdx]); } // ------------- Calculate the Manhattan Distance ---------------- @@ -165,7 +142,7 @@ class ComputeDistanceMapImpl // If the nearestNeighbor value == -1 (invalid value?) and the featureId // of the current voxel is valid. Does this mean we are on a border // voxel like the border between the overscan and sample of an EBSD data set? - if(voxel_NearestNeighbor[i] == -1 && featureIdsStore[i] > 0) + if(voxel_NearestNeighbor[i] == -1 && m_FeatureIds[i] > 0) { count++; // increment the count? // Loop over all neighbors (6 face neighbors) @@ -191,9 +168,9 @@ class ComputeDistanceMapImpl } // Now run back over all voxels to increment "changed" and voxel_Distance. - for(size_t voxelIdx = 0; voxelIdx < totalVoxels; ++voxelIdx) + for(usize voxelIdx = 0; voxelIdx < m_TotalVoxels; ++voxelIdx) { - if(voxel_NearestNeighbor[voxelIdx] != -1 && voxel_Distance[voxelIdx] == -1.0 && featureIdsStore[voxelIdx] > 0) + if(voxel_NearestNeighbor[voxelIdx] != -1 && voxel_Distance[voxelIdx] == -1.0 && m_FeatureIds[voxelIdx] > 0) { changed++; voxel_Distance[voxelIdx] = Distance; @@ -216,14 +193,14 @@ class ComputeDistanceMapImpl yStride = n * xpoints; for(int64_t p = 0; p < xpoints; p++) { - x1 = static_cast(p) * spacing[0]; - y1 = static_cast(n) * spacing[1]; - z1 = static_cast(m) * spacing[2]; + x1 = static_cast(p) * m_Spacing[0]; + y1 = static_cast(n) * m_Spacing[1]; + z1 = static_cast(m) * m_Spacing[2]; if(int64_t nearestNeighbor = voxel_NearestNeighbor[zStride + yStride + p]; nearestNeighbor >= 0) { - x2 = spacing[0] * static_cast(nearestNeighbor % xpoints); // find_xcoord(nearestneighbor); - y2 = spacing[1] * static_cast(static_cast(nearestNeighbor * oneOverxpoints) % ypoints); // find_ycoord(nearestneighbor); - z2 = spacing[2] * floor(nearestNeighbor * oneOverzBlock); // find_zcoord(nearestneighbor); + x2 = m_Spacing[0] * static_cast(nearestNeighbor % xpoints); // find_xcoord(nearestneighbor); + y2 = m_Spacing[1] * static_cast(static_cast(nearestNeighbor * oneOverxpoints) % ypoints); // find_ycoord(nearestneighbor); + z2 = m_Spacing[2] * floor(nearestNeighbor * oneOverzBlock); // find_zcoord(nearestneighbor); dist = ((x1 - x2) * (x1 - x2)) + ((y1 - y2) * (y1 - y2)) + ((z1 - z2) * (z1 - z2)); dist = sqrt(dist); voxel_Distance[zStride + yStride + p] = dist; @@ -233,22 +210,15 @@ class ComputeDistanceMapImpl } } - for(size_t a = 0; a < totalVoxels; ++a) + // Write results back to the nearestNeighbors vector and output distance buffer + for(usize a = 0; a < m_TotalVoxels; ++a) { m_NearestNeighbors[a * 3 + static_cast(MapType)] = voxel_NearestNeighbor[a]; - if(m_InputValues.DoBoundaries && MapType == ComputeEuclideanDistMap::MapType::FeatureBoundary) - { - (*gbManhattanDistancesStore)[a] = static_cast(voxel_Distance[a]); - } - else if(m_InputValues.DoTripleLines && MapType == ComputeEuclideanDistMap::MapType::TripleJunction) - { - (*tjManhattanDistancesStore)[a] = static_cast(voxel_Distance[a]); - } - else if(m_InputValues.DoQuadPoints && MapType == ComputeEuclideanDistMap::MapType::QuadPoint) - { - (*qpManhattanDistancesStore)[a] = static_cast(voxel_Distance[a]); - } + m_DistBuf[a] = static_cast(voxel_Distance[a]); } + + // Bulk-write the distance buffer back to the output DataStore + m_OutputStore->copyFromBuffer(0, nonstd::span(m_DistBuf, m_TotalVoxels)); } }; } // namespace @@ -268,13 +238,17 @@ ComputeEuclideanDistMap::~ComputeEuclideanDistMap() noexcept = default; // ----------------------------------------------------------------------------- template -void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMapInputValues* inputValues) +void FindDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMapInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& messageHandler) { using DataArrayType = DataArray; using DataStoreType = AbstractDataStore; - const auto& featureIdsStore = dataStructure.getDataRefAs(inputValues->FeatureIdsArrayPath).getDataStoreRef(); - size_t totalVoxels = featureIdsStore.getNumberOfTuples(); + const auto& featureIdsStoreRef = dataStructure.getDataRefAs(inputValues->FeatureIdsArrayPath).getDataStoreRef(); + usize totalVoxels = featureIdsStoreRef.getNumberOfTuples(); + + // Bulk-read featureIds into a local buffer to avoid per-element OOC access + std::vector featureIdsBuf(totalVoxels); + featureIdsStoreRef.copyIntoBuffer(0, nonstd::span(featureIdsBuf.data(), totalVoxels)); DataStoreType* gbManhattanDistancesStore = nullptr; if(inputValues->DoBoundaries) @@ -297,6 +271,26 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap qpManhattanDistancesStore->fill(static_cast(-1)); } + // Bulk-read distance stores into local buffers after the fill(-1) call + std::vector gbDistBuf; + if(inputValues->DoBoundaries) + { + gbDistBuf.resize(totalVoxels); + gbManhattanDistancesStore->copyIntoBuffer(0, nonstd::span(gbDistBuf.data(), totalVoxels)); + } + std::vector tjDistBuf; + if(inputValues->DoTripleLines) + { + tjDistBuf.resize(totalVoxels); + tjManhattanDistancesStore->copyIntoBuffer(0, nonstd::span(tjDistBuf.data(), totalVoxels)); + } + std::vector qpDistBuf; + if(inputValues->DoQuadPoints) + { + qpDistBuf.resize(totalVoxels); + qpManhattanDistancesStore->copyIntoBuffer(0, nonstd::span(qpDistBuf.data(), totalVoxels)); + } + // Create a temporary nearest neighbors vector (3 components per voxel) std::vector nearestNeighbors(totalVoxels * 3, -1); @@ -318,9 +312,13 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap // This entire loop finds all 3 kinds of grain boundaries, // Feature Boundaries, Triple Junctions, QuadPoints - for(int64 voxelIndex = 0; voxelIndex < totalVoxels; ++voxelIndex) + for(int64 voxelIndex = 0; voxelIndex < static_cast(totalVoxels); ++voxelIndex) { - feature = featureIdsStore[voxelIndex]; + if(shouldCancel) + { + return; + } + feature = featureIdsBuf[voxelIndex]; if(feature > 0) // Ignore FeatureId = 0 { int64 xIdx = voxelIndex % dims[0]; @@ -341,7 +339,7 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap // If we are a proper neighbor voxel, i.e., have not stepped out of the virtual volume, // and the featureId of the neighbor is NOT the currentFeatureId AND the // neighborFeatureId is valid (greater than 0), then drop into this conditional - if(featureIdsStore[neighborPoint] != feature && featureIdsStore[neighborPoint] >= 0) + if(featureIdsBuf[neighborPoint] != feature && featureIdsBuf[neighborPoint] >= 0) { add = true; // Default to always adding this neighbor to the coordination vector // Loop over the current vector of coordination values @@ -349,7 +347,7 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap { // If the featureId of the neighbor voxel == the current coordination_value // then we set the boolean to ignore this neighbor by setting `add = false` - if(featureIdsStore[neighborPoint] == coordination_value) + if(featureIdsBuf[neighborPoint] == coordination_value) { add = false; break; @@ -357,7 +355,7 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap } if(add) { - coordination.push_back(featureIdsStore[neighborPoint]); // Push back the first neighbor found + coordination.push_back(featureIdsBuf[neighborPoint]); // Push back the first neighbor found } } } @@ -375,9 +373,9 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap // is a grain boundary. Initialize the first component of the nearestNeighbor to the // first value of the coordination vector. // Initialize the GB output array to 0 - if(!coordination.empty() && inputValues->DoBoundaries && nullptr != gbManhattanDistancesStore) + if(!coordination.empty() && inputValues->DoBoundaries) { - (*gbManhattanDistancesStore)[voxelIndex] = 0; + gbDistBuf[voxelIndex] = 0; nearestNeighbors[voxelIndex * 3 + 0] = coordination[0]; nearestNeighbors[voxelIndex * 3 + 1] = -1; nearestNeighbors[voxelIndex * 3 + 2] = -1; @@ -386,9 +384,9 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap // Triple lines are defined as a line that separates 3, and only 3, grains. // Initialize the nearestNeighbor components 0 and 1 to the first value in the coordination vector // Initializes the TJ output array to 0; - if(coordination.size() >= 2 && inputValues->DoTripleLines && nullptr != tjManhattanDistancesStore) + if(coordination.size() >= 2 && inputValues->DoTripleLines) { - (*tjManhattanDistancesStore)[voxelIndex] = 0; + tjDistBuf[voxelIndex] = 0; nearestNeighbors[voxelIndex * 3 + 0] = coordination[0]; nearestNeighbors[voxelIndex * 3 + 1] = coordination[0]; nearestNeighbors[voxelIndex * 3 + 2] = -1; @@ -397,9 +395,9 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap // All other boundaries between 4 or more grains are Quadruple Points. // Initialize the nearestNeighbor components 0, 1, 2 to the first value in the coordination vector // Initializes the QP output array to 0. - if(coordination.size() > 2 && inputValues->DoQuadPoints && nullptr != qpManhattanDistancesStore) + if(coordination.size() > 2 && inputValues->DoQuadPoints) { - (*qpManhattanDistancesStore)[voxelIndex] = 0; + qpDistBuf[voxelIndex] = 0; nearestNeighbors[voxelIndex * 3 + 0] = coordination[0]; nearestNeighbors[voxelIndex * 3 + 1] = coordination[0]; nearestNeighbors[voxelIndex * 3 + 2] = coordination[0]; @@ -408,43 +406,29 @@ void findDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap } } + FloatVec3 spacing = selectedImageGeom.getSpacing(); + // Now that we have all the necessary values, use TBB to initiate a task to compute // the output for each kind of selected output. + // Each task gets its own distance buffer and the shared (read-only) featureIds buffer. + // The template parameter T already encodes the distance type (int32 for Manhattan, float32 for Euclidean). ParallelTaskAlgorithm taskRunner; if(inputValues->DoBoundaries) { - if(inputValues->CalcManhattanDist) - { - taskRunner.execute(ComputeDistanceMapImpl(dataStructure, *inputValues, nearestNeighbors)); - } - else - { - taskRunner.execute(ComputeDistanceMapImpl(dataStructure, *inputValues, nearestNeighbors)); - } + taskRunner.execute(ComputeDistanceMapImpl(*inputValues, nearestNeighbors, featureIdsBuf.data(), gbDistBuf.data(), gbManhattanDistancesStore, + totalVoxels, udims, spacing)); } if(inputValues->DoTripleLines) { - if(inputValues->CalcManhattanDist) - { - taskRunner.execute(ComputeDistanceMapImpl(dataStructure, *inputValues, nearestNeighbors)); - } - else - { - taskRunner.execute(ComputeDistanceMapImpl(dataStructure, *inputValues, nearestNeighbors)); - } + taskRunner.execute(ComputeDistanceMapImpl(*inputValues, nearestNeighbors, featureIdsBuf.data(), tjDistBuf.data(), tjManhattanDistancesStore, + totalVoxels, udims, spacing)); } if(inputValues->DoQuadPoints) { - if(inputValues->CalcManhattanDist) - { - taskRunner.execute(ComputeDistanceMapImpl(dataStructure, *inputValues, nearestNeighbors)); - } - else - { - taskRunner.execute(ComputeDistanceMapImpl(dataStructure, *inputValues, nearestNeighbors)); - } + taskRunner.execute(ComputeDistanceMapImpl(*inputValues, nearestNeighbors, featureIdsBuf.data(), qpDistBuf.data(), qpManhattanDistancesStore, + totalVoxels, udims, spacing)); } // Wait for tasks to complete taskRunner.wait(); @@ -461,11 +445,11 @@ Result<> ComputeEuclideanDistMap::operator()() { if(m_InputValues->CalcManhattanDist) { - findDistanceMap(m_DataStructure, m_InputValues); + FindDistanceMap(m_DataStructure, m_InputValues, m_ShouldCancel, m_MessageHandler); } else { - findDistanceMap(m_DataStructure, m_InputValues); + FindDistanceMap(m_DataStructure, m_InputValues, m_ShouldCancel, m_MessageHandler); } return {}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp index 628e17efca..f742f172cd 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp @@ -5,104 +5,18 @@ #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" #include "simplnx/Utilities/GeometryHelpers.hpp" -#include "simplnx/Utilities/ParallelDataAlgorithm.hpp" + +#include #include +#include +#include using namespace nx::core; namespace { -class ComputeFeatureCentroidsImpl1 -{ -public: - ComputeFeatureCentroidsImpl1(Float64AbstractDataStore& sum, Float64AbstractDataStore& center, UInt64AbstractDataStore& count, std::array dims, const nx::core::ImageGeom& imageGeom, - const Int32AbstractDataStore& featureIds, UInt64AbstractDataStore& rangeXStoreRef, UInt64AbstractDataStore& rangeYStoreRef, UInt64AbstractDataStore& rangeZStoreRef) - : m_Sum(sum) - , m_Center(center) - , m_Count(count) - , m_Dims(dims) - , m_ImageGeom(imageGeom) - , m_FeatureIds(featureIds) - , m_RangeXStoreRef(rangeXStoreRef) - , m_RangeYStoreRef(rangeYStoreRef) - , m_RangeZStoreRef(rangeZStoreRef) - { - } - ~ComputeFeatureCentroidsImpl1() = default; - void compute(usize minFeatureId, usize maxFeatureId) const - { - for(uint64 i = 0; i < m_Dims[2]; i++) - { - size_t zStride = i * m_Dims[0] * m_Dims[1]; - for(uint64 j = 0; j < m_Dims[1]; j++) - { - size_t yStride = j * m_Dims[0]; - for(uint64 k = 0; k < m_Dims[0]; k++) - { - int32 featureId = m_FeatureIds[zStride + yStride + k]; // Get the current FeatureId - if(featureId < minFeatureId || featureId >= maxFeatureId) - { - continue; - } - // Check if feature ID is Periodic - m_RangeXStoreRef[featureId * 2 + 0] = std::min(k, m_RangeXStoreRef.getValue(featureId * 2 + 0)); - m_RangeXStoreRef[featureId * 2 + 1] = std::max(k, m_RangeXStoreRef.getValue(featureId * 2 + 1)); - - m_RangeYStoreRef[featureId * 2 + 0] = std::min(j, m_RangeYStoreRef.getValue(featureId * 2 + 0)); - m_RangeYStoreRef[featureId * 2 + 1] = std::max(j, m_RangeYStoreRef.getValue(featureId * 2 + 1)); - - m_RangeZStoreRef[featureId * 2 + 0] = std::min(i, m_RangeZStoreRef.getValue(featureId * 2 + 0)); - m_RangeZStoreRef[featureId * 2 + 1] = std::max(i, m_RangeZStoreRef.getValue(featureId * 2 + 1)); - - // Get the voxel center based on XYZ index from Image Geom - nx::core::Point3Dd voxel_center = m_ImageGeom.getCoords(k, j, i); - - // Kahan Sum for X Coord - size_t featureId_idx = featureId * 3ULL; - auto componentValue = static_cast(voxel_center[0] - m_Center[featureId_idx]); - double temp = m_Sum[featureId_idx] + componentValue; - m_Center[featureId_idx] = (temp - m_Sum[featureId_idx]) - componentValue; - m_Sum[featureId_idx] = temp; - m_Count[featureId_idx].inc(); - - // Kahan Sum for Y Coord - featureId_idx = featureId * 3ULL + 1; - componentValue = static_cast(voxel_center[1] - m_Center[featureId_idx]); - temp = m_Sum[featureId_idx] + componentValue; - m_Center[featureId_idx] = (temp - m_Sum[featureId_idx]) - componentValue; - m_Sum[featureId_idx] = temp; - m_Count[featureId_idx].inc(); - - // Kahan Sum for Z Coord - featureId_idx = featureId * 3ULL + 2; - componentValue = static_cast(voxel_center[2] - m_Center[featureId_idx]); - temp = m_Sum[featureId_idx] + componentValue; - m_Center[featureId_idx] = (temp - m_Sum[featureId_idx]) - componentValue; - m_Sum[featureId_idx] = temp; - m_Count[featureId_idx].inc(); - } - } - } - } - - void operator()(const Range& range) const - { - compute(range.min(), range.max()); - } - -private: - Float64AbstractDataStore& m_Sum; - Float64AbstractDataStore& m_Center; - UInt64AbstractDataStore& m_Count; - std::array m_Dims = {0, 0, 0}; - const nx::core::ImageGeom& m_ImageGeom; - const Int32AbstractDataStore& m_FeatureIds; - UInt64AbstractDataStore& m_RangeXStoreRef; - UInt64AbstractDataStore& m_RangeYStoreRef; - UInt64AbstractDataStore& m_RangeZStoreRef; -}; - +constexpr usize k_ChunkTuples = 65536; } // namespace // ----------------------------------------------------------------------------- @@ -127,12 +41,9 @@ const std::atomic_bool& ComputeFeatureCentroids::getCancel() // ----------------------------------------------------------------------------- Result<> ComputeFeatureCentroids::operator()() { - // Input Cell Data - const auto* featureIdsPtr = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); const auto& featureIdsStoreRef = featureIdsPtr->getDataStoreRef(); - // Output Feature Data auto& centroids = m_DataStructure.getDataAs(m_InputValues->CentroidsArrayPath)->getDataStoreRef(); auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, m_InputValues->CentroidsArrayPath, *featureIdsPtr, false, m_MessageHandler); @@ -141,75 +52,117 @@ Result<> ComputeFeatureCentroids::operator()() return validateNumFeatResult; } - // Required Geometry const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); + const usize totalFeatures = centroids.getNumberOfTuples(); + const usize xPoints = imageGeom.getNumXCells(); + const usize yPoints = imageGeom.getNumYCells(); + const usize zPoints = imageGeom.getNumZCells(); + + // Plain vectors for accumulation (feature-level, small) to avoid + // AbstractDataStore virtual dispatch in the hot loop. + const usize featureElems3 = totalFeatures * 3; + const usize featureElems2 = totalFeatures * 2; + std::vector kahanSum(featureElems3, 0.0); + std::vector kahanComp(featureElems3, 0.0); + std::vector voxelCount(featureElems3, 0); + std::vector rangeX(featureElems2, 0); + std::vector rangeY(featureElems2, 0); + std::vector rangeZ(featureElems2, 0); + + for(usize f = 0; f < totalFeatures; f++) + { + rangeX[f * 2] = std::numeric_limits::max(); + rangeY[f * 2] = std::numeric_limits::max(); + rangeZ[f * 2] = std::numeric_limits::max(); + } + + const FloatVec3 origin = imageGeom.getOrigin(); + const FloatVec3 spacing = imageGeom.getSpacing(); + const usize totalVoxels = xPoints * yPoints * zPoints; + const usize xySize = xPoints * yPoints; - size_t totalFeatures = centroids.getNumberOfTuples(); - - size_t xPoints = imageGeom.getNumXCells(); - size_t yPoints = imageGeom.getNumYCells(); - size_t zPoints = imageGeom.getNumZCells(); - - ShapeType tupleShape{totalFeatures}; - ShapeType componentShape{3}; - - auto sumPtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); - auto centerPtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); - auto countPtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); - - Float64AbstractDataStore& sum = *sumPtr.get(); - Float64AbstractDataStore& center = *centerPtr.get(); - UInt64AbstractDataStore& count = *countPtr.get(); - - sum.fill(0.0); - center.fill(0.0); - count.fill(0.0); - - // Create data stores to check if feature IDs are periodic - componentShape[0] = 2; - auto rangeXStorePtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); - auto rangeYStorePtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); - auto rangeZStorePtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); - - UInt64AbstractDataStore& rangeXStoreRef = *rangeXStorePtr.get(); - UInt64AbstractDataStore& rangeYStoreRef = *rangeYStorePtr.get(); - UInt64AbstractDataStore& rangeZStoreRef = *rangeZStorePtr.get(); - - // The first part can be expensive so parallelize the algorithm - ParallelDataAlgorithm dataAlg; - dataAlg.setRange(0, totalFeatures); - // This is OFF because we spend more time spinning up threads than actually - // computing things. Maybe if we were to break the total number of features - // by the total number of cores/threads and do a ParallelTask Algorithm instead - // we might see some speedup. - dataAlg.setParallelizationEnabled(false); - dataAlg.execute(ComputeFeatureCentroidsImpl1(sum, center, count, {xPoints, yPoints, zPoints}, imageGeom, featureIdsStoreRef, rangeXStoreRef, rangeYStoreRef, rangeZStoreRef)); - - // Here we are only looping over the number of features so let this just go in serial mode. - for(size_t featureId = 0; featureId < totalFeatures; featureId++) + auto featureIdBuf = std::make_unique(k_ChunkTuples); + for(usize offset = 0; offset < totalVoxels; offset += k_ChunkTuples) { - auto featureId_idx = static_cast(featureId * 3); - if(static_cast(count[featureId_idx]) > 0.0f) + if(m_ShouldCancel) { - centroids[featureId_idx] = static_cast(sum[featureId_idx] / static_cast(count[featureId_idx])); + return {}; } - featureId_idx++; // featureId * 3 + 1 - if(static_cast(count[featureId_idx]) > 0.0f) + const usize chunkCount = std::min(k_ChunkTuples, totalVoxels - offset); + featureIdsStoreRef.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), chunkCount)); + + for(usize idx = 0; idx < chunkCount; idx++) { - centroids[featureId_idx] = static_cast(sum[featureId_idx] / static_cast(count[featureId_idx])); + const int32 featureId = featureIdBuf[idx]; + if(featureId <= 0) + { + continue; + } + + const usize flatIdx = offset + idx; + const uint64 k = flatIdx % xPoints; + const uint64 j = (flatIdx / xPoints) % yPoints; + const uint64 i = flatIdx / xySize; + const usize fid = static_cast(featureId); + + rangeX[fid * 2] = std::min(k, rangeX[fid * 2]); + rangeX[fid * 2 + 1] = std::max(k, rangeX[fid * 2 + 1]); + rangeY[fid * 2] = std::min(j, rangeY[fid * 2]); + rangeY[fid * 2 + 1] = std::max(j, rangeY[fid * 2 + 1]); + rangeZ[fid * 2] = std::min(i, rangeZ[fid * 2]); + rangeZ[fid * 2 + 1] = std::max(i, rangeZ[fid * 2 + 1]); + + const double vx = static_cast(origin[0]) + (static_cast(k) + 0.5) * static_cast(spacing[0]); + const double vy = static_cast(origin[1]) + (static_cast(j) + 0.5) * static_cast(spacing[1]); + const double vz = static_cast(origin[2]) + (static_cast(i) + 0.5) * static_cast(spacing[2]); + + const std::array voxelCoords = {vx, vy, vz}; + for(usize c = 0; c < 3; c++) + { + const usize fi = fid * 3 + c; + const double componentValue = voxelCoords[c] - kahanComp[fi]; + const double temp = kahanSum[fi] + componentValue; + kahanComp[fi] = (temp - kahanSum[fi]) - componentValue; + kahanSum[fi] = temp; + voxelCount[fi]++; + } } + } - featureId_idx++; // featureId * 3 + 2 - if(static_cast(count[featureId_idx]) > 0.0f) + std::vector centroidsBuf(featureElems3, 0.0f); + for(usize featureId = 0; featureId < totalFeatures; featureId++) + { + for(usize c = 0; c < 3; c++) { - centroids[featureId_idx] = static_cast(sum[featureId_idx] / static_cast(count[featureId_idx])); + const usize fi = featureId * 3 + c; + if(voxelCount[fi] > 0) + { + centroidsBuf[fi] = static_cast(kahanSum[fi] / static_cast(voxelCount[fi])); + } } } + centroids.copyFromBuffer(0, nonstd::span(centroidsBuf.data(), featureElems3)); if(m_InputValues->IsPeriodic) { m_MessageHandler({IFilter::Message::Type::Info, "Checking for periodic data."}); + + ShapeType tupleShape{totalFeatures}; + ShapeType componentShape{2}; + auto rangeXStorePtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); + auto rangeYStorePtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); + auto rangeZStorePtr = DataStoreUtilities::CreateDataStore(tupleShape, componentShape, IDataAction::Mode::Execute); + auto& rangeXStoreRef = *rangeXStorePtr; + auto& rangeYStoreRef = *rangeYStorePtr; + auto& rangeZStoreRef = *rangeZStorePtr; + for(usize i = 0; i < featureElems2; i++) + { + rangeXStoreRef[i] = rangeX[i]; + rangeYStoreRef[i] = rangeY[i]; + rangeZStoreRef[i] = rangeZ[i]; + } + if(GeometryHelpers::Topology::AdjustCentroidsForPeriodicFaces(imageGeom, rangeXStoreRef, rangeYStoreRef, rangeZStoreRef, centroids)) { m_MessageHandler({IFilter::Message::Type::Info, "ComputeFeatureCentroids found Non-Contiguous Features. Centroids may require additional checks."}); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp index ada81ad5bf..13c6a1d086 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp @@ -33,7 +33,7 @@ std::vector GenerateRandomDistribution(float32 minDistance, float32 max const float32 maxBoxDistance = sqrtf((boxDims[0] * boxDims[0]) + (boxDims[1] * boxDims[1]) + (boxDims[2] * boxDims[2])); const auto currentNumBins = static_cast(ceil((maxBoxDistance - minDistance) / stepSize)); - freq.resize(static_cast(currentNumBins + 1)); + freq.resize(static_cast(currentNumBins + 1)); std::mt19937_64 generator(userSeedValue); // Standard mersenne_twister_engine seeded std::uniform_real_distribution distribution(0.0, 1.0); @@ -49,9 +49,9 @@ std::vector GenerateRandomDistribution(float32 minDistance, float32 max const usize row = (featureOwnerIdx / xPoints) % yPoints; const usize plane = featureOwnerIdx / (xPoints * yPoints); - const auto xc = static_cast(column * boxRes[0]); - const auto yc = static_cast(row * boxRes[1]); - const auto zc = static_cast(plane * boxRes[2]); + const auto xc = static_cast(column * boxRes[0]); + const auto yc = static_cast(row * boxRes[1]); + const auto zc = static_cast(plane * boxRes[2]); randomCentroids[3 * i] = xc; randomCentroids[3 * i + 1] = yc; @@ -61,13 +61,13 @@ std::vector GenerateRandomDistribution(float32 minDistance, float32 max distanceList.resize(largeNumber); // Calculating all the distances and storing them in the distance list - for(size_t i = 1; i < largeNumber; i++) + for(usize i = 1; i < largeNumber; i++) { const float32 x = randomCentroids[3 * i]; const float32 y = randomCentroids[3 * i + 1]; const float32 z = randomCentroids[3 * i + 2]; - for(size_t j = i + 1; j < largeNumber; j++) + for(usize j = i + 1; j < largeNumber; j++) { const float32 xn = randomCentroids[3 * j]; @@ -99,7 +99,7 @@ std::vector GenerateRandomDistribution(float32 minDistance, float32 max } // Normalize the frequencies - for(size_t i = 0; i < currentNumBins + 1; i++) + for(usize i = 0; i < currentNumBins + 1; i++) { freq[i] /= numDistances; } @@ -131,12 +131,25 @@ const std::atomic_bool& ComputeFeatureClustering::getCancel() Result<> ComputeFeatureClustering::operator()() { const auto& imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); - const auto& featurePhasesStore = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); - const auto& centroidsStore = m_DataStructure.getDataAs(m_InputValues->CentroidsArrayPath)->getDataStoreRef(); + const auto& featurePhasesStoreRef = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); + const auto& centroidsStoreRef = m_DataStructure.getDataAs(m_InputValues->CentroidsArrayPath)->getDataStoreRef(); + + // Cache feature-level arrays locally to avoid per-element OOC overhead in O(n^2) loops + const usize numPhases = featurePhasesStoreRef.getSize(); + std::vector featurePhasesCache(numPhases); + featurePhasesStoreRef.copyIntoBuffer(0, nonstd::span(featurePhasesCache.data(), numPhases)); + + const usize numCentroidValues = centroidsStoreRef.getSize(); + std::vector centroidsCache(numCentroidValues); + centroidsStoreRef.copyIntoBuffer(0, nonstd::span(centroidsCache.data(), numCentroidValues)); auto& clusteringList = m_DataStructure.getDataRefAs>(m_InputValues->ClusteringListArrayName); auto& rdfStore = m_DataStructure.getDataAs(m_InputValues->RDFArrayName)->getDataStoreRef(); auto& minMaxDistancesStore = m_DataStructure.getDataAs(m_InputValues->MaxMinArrayName)->getDataStoreRef(); + + // Cache rdf output array locally + const usize rdfSize = rdfStore.getSize(); + std::vector rdfCache(rdfSize, 0.0f); std::unique_ptr maskCompare; if(m_InputValues->RemoveBiasedFeatures) { @@ -166,7 +179,7 @@ Result<> ComputeFeatureClustering::operator()() std::vector oldCount(m_InputValues->NumberOfBins); std::vector randomRDF; - const usize totalFeatures = featurePhasesStore.getNumberOfTuples(); + const usize totalFeatures = numPhases; SizeVec3 dims = imageGeometry.getDimensions(); FloatVec3 spacing = imageGeometry.getSpacing(); @@ -181,7 +194,7 @@ Result<> ComputeFeatureClustering::operator()() for(usize i = 1; i < totalFeatures; i++) { - if(featurePhasesStore[i] == m_InputValues->PhaseNumber) + if(featurePhasesCache[i] == m_InputValues->PhaseNumber) { totalPptFeatures++; } @@ -195,24 +208,24 @@ Result<> ComputeFeatureClustering::operator()() { return {}; } - if(featurePhasesStore[i] == m_InputValues->PhaseNumber) + if(featurePhasesCache[i] == m_InputValues->PhaseNumber) { if(i % 1000 == 0) { m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Working on Feature {} of {}", i, totalPptFeatures)); } - x = centroidsStore[3 * i]; - y = centroidsStore[3 * i + 1]; - z = centroidsStore[3 * i + 2]; + x = centroidsCache[3 * i]; + y = centroidsCache[3 * i + 1]; + z = centroidsCache[3 * i + 2]; for(usize j = i + 1; j < totalFeatures; j++) { - if(featurePhasesStore[i] == featurePhasesStore[j]) + if(featurePhasesCache[i] == featurePhasesCache[j]) { - xn = centroidsStore[3 * j]; - yn = centroidsStore[3 * j + 1]; - zn = centroidsStore[3 * j + 2]; + xn = centroidsCache[3 * j]; + yn = centroidsCache[3 * j + 1]; + zn = centroidsCache[3 * j + 2]; r = sqrtf((x - xn) * (x - xn) + (y - yn) * (y - yn) + (z - zn) * (z - zn)); @@ -229,7 +242,7 @@ Result<> ComputeFeatureClustering::operator()() { return {}; } - if(featurePhasesStore[i] == m_InputValues->PhaseNumber) + if(featurePhasesCache[i] == m_InputValues->PhaseNumber) { for(auto value : clusters[i]) { @@ -258,7 +271,7 @@ Result<> ComputeFeatureClustering::operator()() { return {}; } - if(featurePhasesStore[i] == m_InputValues->PhaseNumber) + if(featurePhasesCache[i] == m_InputValues->PhaseNumber) { if(maskCompare->isTrue(i)) { @@ -267,13 +280,13 @@ Result<> ComputeFeatureClustering::operator()() for(usize j = 0; j < clusters[i].size(); j++) { - ensemble = featurePhasesStore[i]; + ensemble = featurePhasesCache[i]; bin = (clusters[i][j] - min) / stepSize; if(bin >= m_InputValues->NumberOfBins) { bin = m_InputValues->NumberOfBins - 1; } - rdfStore[(m_InputValues->NumberOfBins * ensemble) + bin].inc(); + rdfCache[(m_InputValues->NumberOfBins * ensemble) + bin]++; } } } @@ -286,17 +299,17 @@ Result<> ComputeFeatureClustering::operator()() { return {}; } - if(featurePhasesStore[i] == m_InputValues->PhaseNumber) + if(featurePhasesCache[i] == m_InputValues->PhaseNumber) { for(usize j = 0; j < clusters[i].size(); j++) { - ensemble = featurePhasesStore[i]; + ensemble = featurePhasesCache[i]; bin = (clusters[i][j] - min) / stepSize; if(bin >= m_InputValues->NumberOfBins) { bin = m_InputValues->NumberOfBins - 1; } - rdfStore[(m_InputValues->NumberOfBins * ensemble) + bin].inc(); + rdfCache[(m_InputValues->NumberOfBins * ensemble) + bin]++; } } } @@ -316,10 +329,13 @@ Result<> ComputeFeatureClustering::operator()() for(usize i = 0; i < m_InputValues->NumberOfBins; i++) { - oldCount[i] = rdfStore[(m_InputValues->NumberOfBins * m_InputValues->PhaseNumber) + i]; - rdfStore[(m_InputValues->NumberOfBins * m_InputValues->PhaseNumber) + i] = oldCount[i] / randomRDF[i + 1]; + oldCount[i] = rdfCache[(m_InputValues->NumberOfBins * m_InputValues->PhaseNumber) + i]; + rdfCache[(m_InputValues->NumberOfBins * m_InputValues->PhaseNumber) + i] = oldCount[i] / randomRDF[i + 1]; } + // Write cached rdf data back to the OOC store + rdfStore.copyFromBuffer(0, nonstd::span(rdfCache.data(), rdfSize)); + clusteringList.setLists(clusters); // for(usize i = 1; i < totalFeatures; i++) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp new file mode 100644 index 0000000000..f571f25996 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp @@ -0,0 +1,29 @@ +#include "ComputeFeatureNeighbors.hpp" + +#include "ComputeFeatureNeighborsDirect.hpp" +#include "ComputeFeatureNeighborsScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighbors::ComputeFeatureNeighbors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeFeatureNeighborsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighbors::~ComputeFeatureNeighbors() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> ComputeFeatureNeighbors::operator()() +{ + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath); + return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp new file mode 100644 index 0000000000..810e3a4dd5 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp @@ -0,0 +1,56 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ArraySelectionParameter.hpp" +#include "simplnx/Parameters/AttributeMatrixSelectionParameter.hpp" +#include "simplnx/Parameters/BoolParameter.hpp" +#include "simplnx/Parameters/DataObjectNameParameter.hpp" +#include "simplnx/Parameters/GeometrySelectionParameter.hpp" + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT ComputeFeatureNeighborsInputValues +{ + DataPath BoundaryCellsPath; + AttributeMatrixSelectionParameter::ValueType CellFeatureArrayPath; + ArraySelectionParameter::ValueType FeatureIdsPath; + GeometrySelectionParameter::ValueType InputImageGeometryPath; + DataPath NeighborListPath; + DataPath NumberOfNeighborsPath; + DataPath SharedSurfaceAreaListPath; + BoolParameter::ValueType StoreBoundaryCells; + BoolParameter::ValueType StoreSurfaceFeatures; + DataPath SurfaceFeaturesPath; +}; + +/** + * @class ComputeFeatureNeighbors + * @brief This algorithm implements support code for the ComputeFeatureNeighborsFilter + */ + +class SIMPLNXCORE_EXPORT ComputeFeatureNeighbors +{ +public: + ComputeFeatureNeighbors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureNeighborsInputValues* inputValues); + ~ComputeFeatureNeighbors() noexcept; + + ComputeFeatureNeighbors(const ComputeFeatureNeighbors&) = delete; + ComputeFeatureNeighbors(ComputeFeatureNeighbors&&) noexcept = delete; + ComputeFeatureNeighbors& operator=(const ComputeFeatureNeighbors&) = delete; + ComputeFeatureNeighbors& operator=(ComputeFeatureNeighbors&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp index dbe75e0b50..d67cb604a2 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp @@ -1,223 +1,640 @@ #include "ComputeFeatureNeighborsDirect.hpp" -#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "ComputeFeatureNeighbors.hpp" + #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/DataStructure/NeighborList.hpp" -#include "simplnx/Utilities/MessageHelper.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" -#include - using namespace nx::core; -// ----------------------------------------------------------------------------- -ComputeFeatureNeighborsDirect::ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - ComputeFeatureNeighborsInputValues* inputValues) -: m_DataStructure(dataStructure) -, m_InputValues(inputValues) -, m_ShouldCancel(shouldCancel) -, m_MessageHandler(mesgHandler) +namespace { -} - -// ----------------------------------------------------------------------------- -ComputeFeatureNeighborsDirect::~ComputeFeatureNeighborsDirect() noexcept = default; - -// ----------------------------------------------------------------------------- -Result<> ComputeFeatureNeighborsDirect::operator()() +// ============================================================================= +// ImageDimensionState — compile-time specialization for image geometry variants. +// +// Encodes which dimensions have only 1 cell ("empty") to enable constexpr +// elimination of boundary processing code for degenerate dimensions. +// Authored by Nathan Young as part of the ComputeFeatureNeighbors rewrite. +// ============================================================================= +template +struct ImageDimensionState { - auto storeBoundaryCells = m_InputValues->StoreBoundaryCells; - auto storeSurfaceFeatures = m_InputValues->StoreSurfaceFeatures; - auto imageGeomPath = m_InputValues->InputImageGeometryPath; - auto featureIdsPath = m_InputValues->FeatureIdsPath; - auto boundaryCellsName = m_InputValues->BoundaryCellsName; - auto numNeighborsName = m_InputValues->NumberOfNeighborsName; - auto neighborListName = m_InputValues->NeighborListName; - auto sharedSurfaceAreaName = m_InputValues->SharedSurfaceAreaListName; - auto surfaceFeaturesName = m_InputValues->SurfaceFeaturesName; - auto featureAttrMatrixPath = m_InputValues->CellFeatureArrayPath; - - DataPath boundaryCellsPath = featureIdsPath.replaceName(boundaryCellsName); - DataPath numNeighborsPath = featureAttrMatrixPath.createChildPath(numNeighborsName); - DataPath neighborListPath = featureAttrMatrixPath.createChildPath(neighborListName); - DataPath sharedSurfaceAreaPath = featureAttrMatrixPath.createChildPath(sharedSurfaceAreaName); - DataPath surfaceFeaturesPath = featureAttrMatrixPath.createChildPath(surfaceFeaturesName); - - auto& featureIds = m_DataStructure.getDataAs(featureIdsPath)->getDataStoreRef(); - auto& numNeighbors = m_DataStructure.getDataAs(numNeighborsPath)->getDataStoreRef(); - - auto& neighborList = m_DataStructure.getDataRefAs(neighborListPath); - auto& sharedSurfaceAreaList = m_DataStructure.getDataRefAs(sharedSurfaceAreaPath); - - auto* boundaryCells = storeBoundaryCells ? m_DataStructure.getDataAs(boundaryCellsPath)->getDataStore() : nullptr; - auto* surfaceFeatures = storeSurfaceFeatures ? m_DataStructure.getDataAs(surfaceFeaturesPath)->getDataStore() : nullptr; - - usize totalPoints = featureIds.getNumberOfTuples(); - usize totalFeatures = numNeighbors.getNumberOfTuples(); + static constexpr bool HasEmptyXDim = EmptyXV; + static constexpr bool HasEmptyYDim = EmptyYV; + static constexpr bool HasEmptyZDim = EmptyZV; - /* Ensure that we will be able to work with the user selected featureId Array */ - const auto [minFeatureId, maxFeatureId] = std::minmax_element(featureIds.begin(), featureIds.end()); - if(static_cast(*maxFeatureId) >= totalFeatures) + static constexpr bool Is1DImageDimsState() { - std::stringstream out; - out << "Data Array " << featureIdsPath.getTargetName() << " has a maximum value of " << *maxFeatureId << " which is greater than the " - << " number of features from array " << numNeighborsPath.getTargetName() << " which has " << totalFeatures << ". Did you select the " - << " incorrect array for the 'FeatureIds' array?"; - return MakeErrorResult(-24500, out.str()); + return (HasEmptyXDim == true && HasEmptyYDim == true && HasEmptyZDim == false) || (HasEmptyXDim == true && HasEmptyYDim == false && HasEmptyZDim == true) || + (HasEmptyXDim == false && HasEmptyYDim == true && HasEmptyZDim == true); } - auto& imageGeom = m_DataStructure.getDataRefAs(imageGeomPath); - SizeVec3 uDims = imageGeom.getDimensions(); + static constexpr bool Is2DImageDimsState() + { + return (HasEmptyXDim == true && HasEmptyYDim == false && HasEmptyZDim == false) || (HasEmptyXDim == false && HasEmptyYDim == true && HasEmptyZDim == false) || + (HasEmptyXDim == false && HasEmptyYDim == false && HasEmptyZDim == true); + } +}; + +using Image3D = ImageDimensionState; +using EmptyXImage2D = ImageDimensionState; +using EmptyYImage2D = ImageDimensionState; +using EmptyZImage2D = ImageDimensionState; +using XImage1D = ImageDimensionState; +using YImage1D = ImageDimensionState; +using ZImage1D = ImageDimensionState; +using SingleVoxelImage = ImageDimensionState; + +template +constexpr bool IsExpectedImageDimsState() +{ + return ActualT::HasEmptyXDim == ExpectedT::HasEmptyXDim && ActualT::HasEmptyYDim == ExpectedT::HasEmptyYDim && ActualT::HasEmptyZDim == ExpectedT::HasEmptyZDim; +} - std::array dims = { - static_cast(uDims[0]), - static_cast(uDims[1]), - static_cast(uDims[2]), - }; +// ============================================================================= +// ComputeFeatureNeighborsFunctor — core algorithm functor. +// +// Template parameters: +// ProcessSurfaceFeaturesV — whether to populate the SurfaceFeatures array +// ProcessBoundaryCellsV — whether to populate the BoundaryCells array +// +// Uses two-stage processing to minimize runtime branching in the innermost loop: +// Stage 1: Boundary cells (corners, edges, faces) — with validity checks +// Stage 2: Internal cells (3D only) — all 6 neighbors guaranteed valid +// +// Surface area accumulation uses per-face area values (computeFaceSurfaceAreas) +// instead of a uniform area, fixing a bug present in DREAM3D 6.5 where all +// faces were assumed to have the same area as the XY face. +// +// Authored by Nathan Young and Jared Duffey. +// ============================================================================= +template +struct ComputeFeatureNeighborsFunctor +{ + template + Result<> operator()(BoolAbstractDataStore* surfaceFeatures, Int8AbstractDataStore* boundaryCells, Float32NeighborList& sharedSurfaceAreaList, Int32NeighborList& neighborsList, + Int32AbstractDataStore& numNeighbors, const Int32AbstractDataStore& featureIds, usize totalFeatures, const std::array& dims, const std::array spacing, + const std::array& neighborVoxelIndexOffsets, const std::atomic_bool& shouldCancel) const + { + if(ProcessSurfaceFeaturesV) + { + if(surfaceFeatures == nullptr) + { + return MakeErrorResult(-789620, "Process Surface Features selected, but the supplied Surface Features Array invalid."); + } + } - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); + if(ProcessBoundaryCellsV) + { + if(boundaryCells == nullptr) + { + return MakeErrorResult(-789621, "Process Boundary Cells selected, but the supplied Boundary Cells Array invalid."); + } + } + + const usize totalPoints = featureIds.getNumberOfTuples(); + + const std::array precomputedFaceAreas = computeFaceSurfaceAreas(spacing); + std::vector> neighborSurfaceAreas(totalFeatures); + + /** + * Stage 1: Process Boundary Cells + * + * The primary goal of Stage 1 is to isolate border cell specific checks out of Phase 2 + * (the internal cells). This includes flagging the border cells without needing to + * branch, and ignoring invalid voxel faces inherently as much as possible. This segmentation + * also allows for removing a branch in the deepest nested loop in Phase 2. + * + * Stage 1 has been split into 3 parts, the vertex (corner), edge, and face cells. + * Of these parts there are two main logic flows defined by `processFrameCell` and + * `processFaceCell`, the main difference between the two being that the "Frame" algorithm + * checks every face neighbor and validates them, whereas the "Face" algorithm removes the + * validation check and cuts down the checked faces to only the valid ones. It should also be + * noted that, optimization is being left on the table with the frame section. It could be further + * broken down into processing each edge/voxel individually to mirror the optimization done to + * faces, but the segmentation done here would make it far less readable and in the greater context + * the speed gain is minimal considering they are O(n-2) and O(1) respectively and the greater algorithm + * is 0(6(n-2)^3). + * + * Note here that discussions were had of adding Kahan Summation for calculating the surface + * areas, but was decided against to conserve memory. At least until the issue presents itself + * in a real world dataset. + */ + const std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); + const auto processFrameCell = [&](const int64 zIndex, const int64 yIndex, const int64 xIndex) -> void { + int8 numDiffNeighbors = 0; + + const int64 voxelIndex = (dims[0] * dims[1] * zIndex) + (dims[0] * yIndex) + xIndex; + const int32 feature = featureIds.getValue(voxelIndex); + if(feature > 0) + { + if constexpr(ProcessSurfaceFeaturesV && !ImageDimensionStateT::Is1DImageDimsState()) + { + surfaceFeatures->setValue(feature, true); + } - int32 feature = 0; - int32 nnum = 0; - uint8 onsurf = 0; + // Loop over the 6 face neighbors of the voxel + std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIndex, yIndex, zIndex, dims); + for(const auto faceIndex : faceNeighborInternalIdx) // ref more expensive than trivial copy for scalar types + { + if(!isValidFaceNeighbor[faceIndex]) + { + continue; + } + + const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + + const int32 neighborFeatureId = featureIds.getValue(neighborPoint); + if(neighborFeatureId != feature && neighborFeatureId > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeatureId] += precomputedFaceAreas[faceIndex]; + } + } + } + if constexpr(ProcessBoundaryCellsV) + { + boundaryCells->setValue(voxelIndex, numDiffNeighbors); + } + }; - std::vector> neighborlist(totalFeatures); - std::vector> neighborsurfacearealist(totalFeatures); + // Process Corners + { + /** + * Process Corners: + * The constexpr logic in the code block will handle the following, using XYZ indexes: + * + * Case 1: Empty X + * - 0,0,0 + * - 0,n_Y,0 + * - 0,0,n_Z + * - 0,n_Y,n_Z + * + * Case 2: Empty Y + * - 0,0,0 + * - n_X,0,0 + * - 0,0,n_Z + * - n_X,0,n_Z + * + * Case 3: Empty Z + * - 0,0,0 + * - n_X,0,0 + * - 0,n_Y,0 + * - n_X,n_Y,0 + * + * Case 4: 3D Image (Image Stack) + * - 0,0,0 + * - n_X,0,0 + * - 0,n_Y,0 + * - 0,0,n_Z + * - n_X,n_Y,0 + * - n_X,0,n_Z + * - 0,n_Y,n_Z + * - n_X,n_Y,n_Z + */ + + processFrameCell(0, 0, 0); + if constexpr(ProcessSurfaceFeaturesV && ImageDimensionStateT::Is1DImageDimsState()) + { + // Since the frame cell function is shared between corners and edges + // 1D case for border feature flagging must be disabled to prevent + // the entire row from being flagged, thus we must do the corners + // in an explicit action. + // Note here that there is an argument that a new function + // should be defined to handle corner cells. However, this was + // decided against to avoid needles code duplication, as the + // difference between the two functions would be a single + // constexpr if statement + const int32 feature = featureIds.getValue(0); + surfaceFeatures->setValue(feature, true); + } + if constexpr(!IsExpectedImageDimsState()) + { + processFrameCell(dims[2] - 1, dims[1] - 1, dims[0] - 1); // If 2D the dims in empty dimension is 1 so this line effectively preforms for all cases - int32 nListSize = 100; + if constexpr(ProcessSurfaceFeaturesV && ImageDimensionStateT::Is1DImageDimsState()) + { + // Since the frame cell function is shared between corners and edges + // 1D case for border feature flagging must be disabled to prevent + // the entire row from being flagged, thus we must do the corners + // in an explicit action. + // Note here that there is an argument that a new function + // should be defined to handle corner cells. However, this was + // decided against to avoid needles code duplication, as the + // difference between the two functions would be a single + // constexpr if statement + const int64 voxelIndex = (dims[0] * dims[1] * (dims[2] - 1)) + (dims[0] * (dims[1] - 1)) + (dims[0] - 1); + const int32 feature = featureIds.getValue(voxelIndex); + surfaceFeatures->setValue(feature, true); + } - MessageHelper messageHelper(m_MessageHandler); - ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); - // Initialize the neighbor lists - for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) - { - auto now = std::chrono::steady_clock::now(); - throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Initializing Neighbor Lists || {:.2f}% Complete", CalculatePercentComplete(featureIdx, totalFeatures)); }); + if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) + { + if constexpr(!IsExpectedImageDimsState()) + { + processFrameCell(0, 0, dims[0] - 1); + } + if constexpr(!IsExpectedImageDimsState()) + { + processFrameCell(0, dims[1] - 1, 0); + } + if constexpr(!IsExpectedImageDimsState()) + { + processFrameCell(dims[2] - 1, 0, 0); + } + if constexpr(IsExpectedImageDimsState()) + { + processFrameCell(0, dims[1] - 1, dims[0] - 1); + processFrameCell(dims[2] - 1, 0, dims[0] - 1); + processFrameCell(dims[2] - 1, dims[1] - 1, 0); + } + } + } + } - if(m_ShouldCancel) + // Case 0: Process Edges + // X Edges + if constexpr((ImageDimensionStateT::Is2DImageDimsState() && !IsExpectedImageDimsState()) || IsExpectedImageDimsState() || + IsExpectedImageDimsState()) { - return {}; + for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) + { + processFrameCell(0, 0, xIndex); + if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) + { + if constexpr(IsExpectedImageDimsState()) + { + processFrameCell(0, dims[1] - 1, xIndex); + processFrameCell(dims[2] - 1, 0, xIndex); + } + processFrameCell(dims[2] - 1, dims[1] - 1, xIndex); + } + } } - numNeighbors[featureIdx] = 0; - neighborlist[featureIdx].resize(nListSize); - neighborsurfacearealist[featureIdx].assign(nListSize, -1.0f); - if(storeSurfaceFeatures && surfaceFeatures != nullptr) + // Y Edges + if constexpr((ImageDimensionStateT::Is2DImageDimsState() && !IsExpectedImageDimsState()) || IsExpectedImageDimsState() || + IsExpectedImageDimsState()) { - surfaceFeatures->setValue(featureIdx, false); + for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) + { + processFrameCell(0, yIndex, 0); + if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) + { + if constexpr(IsExpectedImageDimsState()) + { + processFrameCell(0, yIndex, dims[0] - 1); + processFrameCell(dims[2] - 1, yIndex, 0); + } + processFrameCell(dims[2] - 1, yIndex, dims[0] - 1); + } + } } - } - - // Loop over all points to generate the neighbor lists - for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) - { - throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Determining Neighbor Lists || {:.2f}% Complete", CalculatePercentComplete(voxelIndex, totalPoints)); }); - if(m_ShouldCancel) + // Z Edges + if constexpr((ImageDimensionStateT::Is2DImageDimsState() && !IsExpectedImageDimsState()) || IsExpectedImageDimsState() || + IsExpectedImageDimsState()) { - return {}; + for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) + { + processFrameCell(zIndex, 0, 0); + if constexpr(!ImageDimensionStateT::Is1DImageDimsState()) + { + if constexpr(IsExpectedImageDimsState()) + { + processFrameCell(zIndex, 0, dims[0] - 1); + processFrameCell(zIndex, dims[1] - 1, 0); + } + processFrameCell(zIndex, dims[1] - 1, dims[0] - 1); + } + } } - onsurf = 0; - feature = featureIds[voxelIndex]; - if(feature > 0 && feature < neighborlist.size()) + // Process Planes for 2D and 3D (Stack) Images + if constexpr(!ImageDimensionStateT::Is1DImageDimsState() && !IsExpectedImageDimsState()) { - int64 xIdx = voxelIndex % dims[0]; - int64 yIdx = (voxelIndex / dims[0]) % dims[1]; - int64 zIdx = voxelIndex / (dims[0] * dims[1]); + const auto processFaceCell = [&](const int64 zIndex, const int64 yIndex, const int64 xIndex, const std::vector& validFaces) -> void { + int8 numDiffNeighbors = 0; + + const int64 voxelIndex = (dims[0] * dims[1] * zIndex) + (dims[0] * yIndex) + xIndex; + const int32 feature = featureIds.getValue(voxelIndex); + if(feature > 0) + { + if constexpr(ProcessSurfaceFeaturesV && IsExpectedImageDimsState()) + { + surfaceFeatures->setValue(feature, true); + } + + // Loop over the face neighbors of the voxel + for(const auto faceIndex : validFaces) // ref more expensive than trivial copy for scalar types + { + const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + + const int32 neighborFeatureId = featureIds.getValue(neighborPoint); + if(neighborFeatureId != feature && neighborFeatureId > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeatureId] += precomputedFaceAreas[faceIndex]; + } + } + } + if constexpr(ProcessBoundaryCellsV) + { + boundaryCells->setValue(voxelIndex, numDiffNeighbors); + } + }; - if(storeSurfaceFeatures && surfaceFeatures != nullptr) + // Case 1: Z Planes { - if((xIdx == 0 || xIdx == static_cast((dims[0] - 1)) || yIdx == 0 || yIdx == static_cast((dims[1]) - 1) || zIdx == 0 || zIdx == static_cast((dims[2] - 1))) && dims[2] != 1) + if constexpr(IsExpectedImageDimsState()) { - surfaceFeatures->setValue(feature, true); + std::vector negZValidFaces = {k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; + std::vector posZValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor}; + for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) + { + for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) + { + processFaceCell(0, yIndex, xIndex, negZValidFaces); + processFaceCell(dims[2] - 1, yIndex, xIndex, posZValidFaces); + } + } } - if((xIdx == 0 || xIdx == static_cast((dims[0] - 1)) || yIdx == 0 || yIdx == static_cast((dims[1] - 1))) && dims[2] == 1) + if constexpr(IsExpectedImageDimsState()) { - surfaceFeatures->setValue(feature, true); + std::vector validFaces = {k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor}; + for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) + { + for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) + { + processFaceCell(0, yIndex, xIndex, validFaces); + } + } } } - // Loop over the 6 face neighbors of the voxel - std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(const auto& faceIndex : faceNeighborInternalIdx) + // Case 2: Y Planes { - if(!isValidFaceNeighbor[faceIndex]) + if constexpr(IsExpectedImageDimsState()) { - continue; + std::vector negYValidFaces = {k_NegativeZNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; + std::vector posYValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveZNeighbor}; + for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) + { + for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) + { + processFaceCell(zIndex, 0, xIndex, negYValidFaces); + processFaceCell(zIndex, dims[1] - 1, xIndex, posYValidFaces); + } + } } + if constexpr(IsExpectedImageDimsState()) + { + std::vector validFaces = {k_NegativeZNeighbor, k_NegativeXNeighbor, k_PositiveXNeighbor, k_PositiveZNeighbor}; + for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) + { + for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) + { + processFaceCell(zIndex, 0, xIndex, validFaces); + } + } + } + } - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + // Case 3: X Planes + { + if constexpr(IsExpectedImageDimsState()) + { + std::vector negXValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_PositiveXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; + std::vector posXValidFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_NegativeXNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; + for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) + { + for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) + { + processFaceCell(zIndex, yIndex, 0, negXValidFaces); + processFaceCell(zIndex, yIndex, dims[0] - 1, posXValidFaces); + } + } + } + if constexpr(IsExpectedImageDimsState()) + { + std::vector validFaces = {k_NegativeZNeighbor, k_NegativeYNeighbor, k_PositiveYNeighbor, k_PositiveZNeighbor}; + for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) + { + for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) + { + processFaceCell(zIndex, yIndex, 0, validFaces); + } + } + } + } + } - if(featureIds[neighborPoint] != feature && featureIds[neighborPoint] > 0) + /** + * Stage 2: Process Internal Cells + * This stage has a bulk of the computation, and runtime branching has been minimized + * to reflect that reality, see comment for Stage 1. This section just walks every + * internal cell and checks each of the neighbors, storing them onto the existing + * results from the boundary cell phases. + */ + if constexpr(IsExpectedImageDimsState()) + { + // Loop over all internal cells to generate the neighbor lists + for(int64 zIndex = 1; zIndex < dims[2] - 1; zIndex++) + { + const int64 zStride = dims[0] * dims[1] * zIndex; + for(int64 yIndex = 1; yIndex < dims[1] - 1; yIndex++) { - onsurf++; - nnum = numNeighbors[feature]; - neighborlist[feature].push_back(featureIds[neighborPoint]); - nnum++; - numNeighbors[feature] = nnum; + const int64 yStride = dims[0] * yIndex; + if(shouldCancel) + { + return {}; + } + for(int64 xIndex = 1; xIndex < dims[0] - 1; xIndex++) + { + int64 voxelIndex = zStride + yStride + xIndex; + + // This value tracks the number of neighboring cells that have feature ids different from itself + int8 numDiffNeighbors = 0; + int32 feature = featureIds.getValue(voxelIndex); + if(feature > 0) + { + // Loop over the face neighbors of the voxel + for(const auto faceIndex : faceNeighborInternalIdx) // ref more expensive than trivial copy for scalar types + { + // No need for a face validity check because we are only processing internal cells + + const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + + const int32 neighborFeatureId = featureIds.getValue(neighborPoint); + if(neighborFeatureId != feature && neighborFeatureId > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeatureId] += precomputedFaceAreas[faceIndex]; + } + } + } + if constexpr(ProcessBoundaryCellsV) + { + boundaryCells->setValue(voxelIndex, numDiffNeighbors); + } + } } } } - if(storeBoundaryCells && boundaryCells != nullptr) + + for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) { - boundaryCells->setValue(voxelIndex, static_cast(onsurf)); + const usize neighborCount = neighborSurfaceAreas[featureIdx].size(); + numNeighbors.setValue(featureIdx, static_cast(neighborCount)); + + // Set the vector for each list into the NeighborList Object + auto sharedNeiLst = std::make_shared::VectorType>(); + sharedNeiLst->reserve(neighborCount); + auto sharedSAL = std::make_shared::VectorType>(); + sharedSAL->reserve(neighborCount); + for(const auto& [featureId, surfaceArea] : neighborSurfaceAreas[featureIdx]) + { + sharedNeiLst->push_back(static_cast(featureId)); + sharedSAL->push_back(static_cast(surfaceArea)); + } + neighborsList.setList(static_cast(featureIdx), sharedNeiLst); + sharedSurfaceAreaList.setList(static_cast(featureIdx), sharedSAL); } + + return {}; } +}; - FloatVec3 spacing = imageGeom.getSpacing(); +template +Result<> ProcessVoxels(const FunctorT& functor, const ImageGeom& imageGeom, ArgsT&&... args) +{ + const bool xDimEmpty = imageGeom.getNumXCells() == 1; + const bool yDimEmpty = imageGeom.getNumYCells() == 1; + const bool zDimEmpty = imageGeom.getNumZCells() == 1; + const uint8 emptyDimCount = static_cast(xDimEmpty) + static_cast(yDimEmpty) + static_cast(zDimEmpty); - // We do this to create new set of NeighborList objects - for(usize i = 1; i < totalFeatures; i++) + // Treat dimensions of 1 as flat for image geom + if(emptyDimCount == 0) { - throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Calculating Surface Areas || {:.2f}% Complete", CalculatePercentComplete(i, totalFeatures)); }); - - if(m_ShouldCancel) + return functor.template operator()(std::forward(args)...); + } + if(emptyDimCount == 1) + { + if(zDimEmpty) { - return {}; + return functor.template operator()(std::forward(args)...); } - - std::map neighToCount; - auto numneighs = static_cast(neighborlist[i].size()); - - // this increments the voxel counts for each feature - for(int32 j = 0; j < numneighs; j++) + if(yDimEmpty) { - neighToCount[neighborlist[i][j]]++; + return functor.template operator()(std::forward(args)...); } - - auto neighborIter = neighToCount.find(0); - neighToCount.erase(neighborIter); - neighborIter = neighToCount.find(-1); - if(neighborIter != neighToCount.end()) + if(xDimEmpty) { - neighToCount.erase(neighborIter); + return functor.template operator()(std::forward(args)...); } - // Resize the features neighbor list to zero - neighborlist[i].resize(0); - neighborsurfacearealist[i].resize(0); - - for(const auto [neigh, number] : neighToCount) + } + if(emptyDimCount == 2) + { + if(xDimEmpty && yDimEmpty) { - float area = static_cast(number) * spacing[0] * spacing[1]; - - // Push the neighbor feature identifier back onto the list, so we stay synced up - neighborlist[i].push_back(neigh); - neighborsurfacearealist[i].push_back(area); + return functor.template operator()(std::forward(args)...); } - numNeighbors[i] = static_cast(neighborlist[i].size()); + if(xDimEmpty && zDimEmpty) + { + return functor.template operator()(std::forward(args)...); + } + if(yDimEmpty && zDimEmpty) + { + return functor.template operator()(std::forward(args)...); + } + } + if(emptyDimCount == 3) + { + return functor.template operator()(std::forward(args)...); + } - // Set the vector for each list into the NeighborList Object - NeighborList::SharedVectorType sharedNeiLst(new std::vector); - sharedNeiLst->assign(neighborlist[i].begin(), neighborlist[i].end()); - neighborList.setList(static_cast(i), sharedNeiLst); + return {}; +} +} // namespace - NeighborList::SharedVectorType sharedSAL(new std::vector); - sharedSAL->assign(neighborsurfacearealist[i].begin(), neighborsurfacearealist[i].end()); - sharedSurfaceAreaList.setList(static_cast(i), sharedSAL); +// ----------------------------------------------------------------------------- +ComputeFeatureNeighborsDirect::ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeFeatureNeighborsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighborsDirect::~ComputeFeatureNeighborsDirect() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief In-core implementation of ComputeFeatureNeighbors using Nathan Young's + * rewritten algorithm with compile-time dimension specialization and per-face + * surface area accumulation. + * + * Uses getValue() for per-element array access (in-core optimal). + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + */ +Result<> ComputeFeatureNeighborsDirect::operator()() +{ + auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath)->getDataStoreRef(); + auto& numNeighbors = m_DataStructure.getDataAs(m_InputValues->NumberOfNeighborsPath)->getDataStoreRef(); + + auto& neighborsList = m_DataStructure.getDataRefAs(m_InputValues->NeighborListPath); + auto& sharedSurfaceAreaList = m_DataStructure.getDataRefAs(m_InputValues->SharedSurfaceAreaListPath); + + usize totalFeatures = numNeighbors.getNumberOfTuples(); + + /* Ensure that we will be able to work with the user selected featureId Array */ + const int32 maxFeatureId = *std::max_element(featureIds.cbegin(), featureIds.cend()); + if(static_cast(maxFeatureId) >= totalFeatures) + { + std::stringstream out; + out << "Data Array " << m_InputValues->FeatureIdsPath.getTargetName() << " has a maximum value of " << maxFeatureId << " which is greater than the " + << " number of features from array " << m_InputValues->NumberOfNeighborsPath.getTargetName() << " which has " << totalFeatures << ". Did you select the " + << " incorrect array for the 'FeatureIds' array?"; + return MakeErrorResult(-24500, out.str()); } - return {}; + const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometryPath); + SizeVec3 uDims = imageGeom.getDimensions(); + + std::array dims = {static_cast(uDims[0]), static_cast(uDims[1]), static_cast(uDims[2])}; + + FloatVec3 spacing32 = imageGeom.getSpacing(); + + std::array spacing64 = {static_cast(spacing32[0]), static_cast(spacing32[1]), static_cast(spacing32[2])}; + + std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); + + if(m_InputValues->StoreSurfaceFeatures && m_InputValues->StoreBoundaryCells) + { + // Surface Features filled with `false` by default during creation in preflight + auto* surfaceFeatures = m_DataStructure.getDataAs(m_InputValues->SurfaceFeaturesPath)->getDataStore(); + auto* boundaryCells = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsPath)->getDataStore(); + return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, surfaceFeatures, boundaryCells, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, + spacing64, neighborVoxelIndexOffsets, m_ShouldCancel); + } + if(m_InputValues->StoreSurfaceFeatures) + { + // Surface Features filled with `false` by default during creation in preflight + auto* surfaceFeatures = m_DataStructure.getDataAs(m_InputValues->SurfaceFeaturesPath)->getDataStore(); + return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, surfaceFeatures, nullptr, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, + spacing64, neighborVoxelIndexOffsets, m_ShouldCancel); + } + if(m_InputValues->StoreBoundaryCells) + { + auto* boundaryCells = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsPath)->getDataStore(); + return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, nullptr, boundaryCells, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, + spacing64, neighborVoxelIndexOffsets, m_ShouldCancel); + } + + return ProcessVoxels(::ComputeFeatureNeighborsFunctor{}, imageGeom, nullptr, nullptr, sharedSurfaceAreaList, neighborsList, numNeighbors, featureIds, totalFeatures, dims, spacing64, + neighborVoxelIndexOffsets, m_ShouldCancel); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp index c4d9d41770..f6fdfd131f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp @@ -2,58 +2,35 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ArraySelectionParameter.hpp" -#include "simplnx/Parameters/AttributeMatrixSelectionParameter.hpp" -#include "simplnx/Parameters/BoolParameter.hpp" -#include "simplnx/Parameters/DataObjectNameParameter.hpp" -#include "simplnx/Parameters/GeometrySelectionParameter.hpp" - -/** -* This is example code to put in the Execute Method of the filter. - ComputeFeatureNeighborsInputValues inputValues; - inputValues.BoundaryCellsName = filterArgs.value(boundary_cells_name); - inputValues.CellFeatureArrayPath = filterArgs.value(cell_feature_array_path); - inputValues.FeatureIdsPath = filterArgs.value(feature_ids_path); - inputValues.InputImageGeometryPath = filterArgs.value(input_image_geometry_path); - inputValues.NeighborListName = filterArgs.value(neighbor_list_name); - inputValues.NumberOfNeighborsName = filterArgs.value(number_of_neighbors_name); - inputValues.SharedSurfaceAreaListName = filterArgs.value(shared_surface_area_list_name); - inputValues.StoreBoundaryCells = filterArgs.value(store_boundary_cells); - inputValues.StoreSurfaceFeatures = filterArgs.value(store_surface_features); - inputValues.SurfaceFeaturesName = filterArgs.value(surface_features_name); - return ComputeFeatureNeighborsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); - -*/ namespace nx::core { - -struct SIMPLNXCORE_EXPORT ComputeFeatureNeighborsInputValues -{ - DataObjectNameParameter::ValueType BoundaryCellsName; - AttributeMatrixSelectionParameter::ValueType CellFeatureArrayPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - DataObjectNameParameter::ValueType NeighborListName; - DataObjectNameParameter::ValueType NumberOfNeighborsName; - DataObjectNameParameter::ValueType SharedSurfaceAreaListName; - BoolParameter::ValueType StoreBoundaryCells; - BoolParameter::ValueType StoreSurfaceFeatures; - DataObjectNameParameter::ValueType SurfaceFeaturesName; -}; +struct ComputeFeatureNeighborsInputValues; /** * @class ComputeFeatureNeighborsDirect - * @brief This algorithm implements support code for the ComputeFeatureNeighborsFilter + * @brief In-core algorithm for ComputeFeatureNeighbors using compile-time dimension + * specialization and per-face surface area accumulation. + * + * Uses Nathan Young's rewritten algorithm with two-stage processing: + * Stage 1: Boundary cells (corners, edges, faces) with validity checks + * Stage 2: Internal cells (3D only) with all 6 neighbors guaranteed valid + * + * Accumulates per-face surface areas using precomputed face dimensions rather + * than a uniform area, fixing a surface area calculation bug from DREAM3D 6.5. + * Handles 0D/1D/2D/3D geometries via constexpr template specialization. + * + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * + * @see ComputeFeatureNeighborsScanline for the out-of-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ - class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsDirect { public: - ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureNeighborsInputValues* inputValues); + ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeFeatureNeighborsInputValues* inputValues); ~ComputeFeatureNeighborsDirect() noexcept; ComputeFeatureNeighborsDirect(const ComputeFeatureNeighborsDirect&) = delete; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp new file mode 100644 index 0000000000..4c801b8b21 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp @@ -0,0 +1,276 @@ +#include "ComputeFeatureNeighborsScanline.hpp" + +#include "ComputeFeatureNeighbors.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/NeighborList.hpp" +#include "simplnx/Utilities/NeighborUtilities.hpp" + +#include + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighborsScanline::ComputeFeatureNeighborsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeFeatureNeighborsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeFeatureNeighborsScanline::~ComputeFeatureNeighborsScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief Computes feature neighbor lists using Z-slice bulk I/O with per-face + * surface area accumulation. + * + * OOC path: Reads FeatureIds one Z-slice at a time via copyIntoBuffer using a + * 3-slice rolling window (prev/cur/next) to resolve all 6 face neighbors. + * BoundaryCells output is written per-slice via copyFromBuffer. + * + * Uses map-based per-feature surface area accumulation with per-face area values + * (computeFaceSurfaceAreas), matching Nathan Young's bug fix for correct surface + * area computation across faces of different sizes. + * + * Surface feature detection handles 3D (all 6 boundary planes) and 2D (4 boundary + * edges in the non-degenerate plane). 1D and 0D geometries are small enough to + * always use the in-core Direct path, but are handled correctly here as well. + */ +Result<> ComputeFeatureNeighborsScanline::operator()() +{ + auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath)->getDataStoreRef(); + auto& numNeighbors = m_DataStructure.getDataAs(m_InputValues->NumberOfNeighborsPath)->getDataStoreRef(); + + auto& neighborList = m_DataStructure.getDataRefAs(m_InputValues->NeighborListPath); + auto& sharedSurfaceAreaList = m_DataStructure.getDataRefAs(m_InputValues->SharedSurfaceAreaListPath); + + auto* boundaryCellsStore = m_InputValues->StoreBoundaryCells ? m_DataStructure.getDataAs(m_InputValues->BoundaryCellsPath)->getDataStore() : nullptr; + auto* surfaceFeatures = m_InputValues->StoreSurfaceFeatures ? m_DataStructure.getDataAs(m_InputValues->SurfaceFeaturesPath)->getDataStore() : nullptr; + + usize totalFeatures = numNeighbors.getNumberOfTuples(); + + auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometryPath); + SizeVec3 uDims = imageGeom.getDimensions(); + + const int64 dimX = static_cast(uDims[0]); + const int64 dimY = static_cast(uDims[1]); + const int64 dimZ = static_cast(uDims[2]); + const usize sliceSize = static_cast(dimX) * static_cast(dimY); + + FloatVec3 spacing32 = imageGeom.getSpacing(); + std::array spacing64 = {static_cast(spacing32[0]), static_cast(spacing32[1]), static_cast(spacing32[2])}; + + // Per-face areas indexed by FaceNeighborType: + // [0] = -Z face (XY plane), [1] = -Y face (XZ plane), [2] = -X face (YZ plane), + // [3] = +X face (YZ plane), [4] = +Y face (XZ plane), [5] = +Z face (XY plane) + const std::array precomputedFaceAreas = computeFaceSurfaceAreas(spacing64); + + // Map-based accumulation: neighborSurfaceAreas[featureId][neighborFeatureId] = total shared area + // This replaces the old vector-based counting + deduplication approach and fixes the + // surface area calculation bug where all faces were assumed to have the same area. + std::vector> neighborSurfaceAreas(totalFeatures); + + // Max feature ID validation is deferred to after the slice loop + // to avoid a separate full scan through OOC data. The loop's + // `feature < totalFeatures` guard prevents out-of-bounds access. + int32 observedMaxFeatureId = 0; + + // 3-slice rolling window for Z-sequential bulk I/O + std::vector prevSlice(sliceSize); + std::vector curSlice(sliceSize); + std::vector nextSlice(sliceSize); + std::vector boundaryCellsSlice; + if(boundaryCellsStore != nullptr) + { + boundaryCellsSlice.resize(sliceSize, 0); + } + + // Load the first slice + featureIds.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + // Load the second slice if it exists + if(dimZ > 1) + { + featureIds.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + + for(int64 z = 0; z < dimZ; z++) + { + if(m_ShouldCancel) + { + return {}; + } + + if(boundaryCellsStore != nullptr) + { + std::fill(boundaryCellsSlice.begin(), boundaryCellsSlice.end(), static_cast(0)); + } + + for(int64 y = 0; y < dimY; y++) + { + const usize yStride = static_cast(y) * static_cast(dimX); + + for(int64 x = 0; x < dimX; x++) + { + const usize localIndex = yStride + static_cast(x); + int8 numDiffNeighbors = 0; + const int32 feature = curSlice[localIndex]; + + if(feature > observedMaxFeatureId) + { + observedMaxFeatureId = feature; + } + + if(feature > 0 && static_cast(feature) < totalFeatures) + { + // Surface feature detection: a feature is a surface feature if it + // touches a boundary face in a non-degenerate dimension (size > 1). + // Dimensions with size == 1 are "empty" and their boundary faces + // do not count — except when ALL dimensions are degenerate (single + // voxel), in which case the feature is trivially on the surface. + // This matches Nathan Young's constexpr ImageDimensionState + // handling in the Direct variant. + if(surfaceFeatures != nullptr) + { + bool isBoundary = (dimX == 1 && dimY == 1 && dimZ == 1); // single voxel is trivially surface + if(dimX > 1 && (x == 0 || x == dimX - 1)) + { + isBoundary = true; + } + if(dimY > 1 && (y == 0 || y == dimY - 1)) + { + isBoundary = true; + } + if(dimZ > 1 && (z == 0 || z == dimZ - 1)) + { + isBoundary = true; + } + if(isBoundary) + { + surfaceFeatures->setValue(feature, true); + } + } + + // Check -Z neighbor (from previous slice buffer) + if(z > 0) + { + const int32 neighborFeature = prevSlice[localIndex]; + if(neighborFeature != feature && neighborFeature > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeature] += precomputedFaceAreas[k_NegativeZNeighbor]; + } + } + + // Check -Y neighbor (within current slice) + if(y > 0) + { + const int32 neighborFeature = curSlice[localIndex - static_cast(dimX)]; + if(neighborFeature != feature && neighborFeature > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeature] += precomputedFaceAreas[k_NegativeYNeighbor]; + } + } + + // Check -X neighbor (within current slice) + if(x > 0) + { + const int32 neighborFeature = curSlice[localIndex - 1]; + if(neighborFeature != feature && neighborFeature > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeature] += precomputedFaceAreas[k_NegativeXNeighbor]; + } + } + + // Check +X neighbor (within current slice) + if(x < dimX - 1) + { + const int32 neighborFeature = curSlice[localIndex + 1]; + if(neighborFeature != feature && neighborFeature > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeature] += precomputedFaceAreas[k_PositiveXNeighbor]; + } + } + + // Check +Y neighbor (within current slice) + if(y < dimY - 1) + { + const int32 neighborFeature = curSlice[localIndex + static_cast(dimX)]; + if(neighborFeature != feature && neighborFeature > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeature] += precomputedFaceAreas[k_PositiveYNeighbor]; + } + } + + // Check +Z neighbor (from next slice buffer) + if(z < dimZ - 1) + { + const int32 neighborFeature = nextSlice[localIndex]; + if(neighborFeature != feature && neighborFeature > 0) + { + numDiffNeighbors++; + neighborSurfaceAreas[feature][neighborFeature] += precomputedFaceAreas[k_PositiveZNeighbor]; + } + } + } + + if(boundaryCellsStore != nullptr) + { + boundaryCellsSlice[localIndex] = numDiffNeighbors; + } + } + } + + // Write the boundaryCells slice to the output store + if(boundaryCellsStore != nullptr) + { + boundaryCellsStore->copyFromBuffer(static_cast(z) * sliceSize, nonstd::span(boundaryCellsSlice.data(), sliceSize)); + } + + // Rotate the rolling window + std::swap(prevSlice, curSlice); + std::swap(curSlice, nextSlice); + if(z + 2 < dimZ) + { + featureIds.copyIntoBuffer(static_cast(z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + } + + // Validate max feature ID (deferred from before the loop to avoid a separate OOC scan) + if(static_cast(observedMaxFeatureId) >= totalFeatures) + { + return MakeErrorResult(-24500, fmt::format("Data Array {} has a maximum value of {} which is greater than the number of features from array {} which has {}. " + "Did you select the incorrect array for the 'FeatureIds' array?", + m_InputValues->FeatureIdsPath.getTargetName(), observedMaxFeatureId, m_InputValues->NumberOfNeighborsPath.getTargetName(), totalFeatures)); + } + + // Convert accumulated per-feature surface area maps to NeighborList objects. + // Map keys are sorted by neighbor feature ID, matching the Direct variant's output order. + for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) + { + const usize neighborCount = neighborSurfaceAreas[featureIdx].size(); + numNeighbors.setValue(featureIdx, static_cast(neighborCount)); + + auto sharedNeiLst = std::make_shared::VectorType>(); + sharedNeiLst->reserve(neighborCount); + auto sharedSAL = std::make_shared::VectorType>(); + sharedSAL->reserve(neighborCount); + for(const auto& [featureId, surfaceArea] : neighborSurfaceAreas[featureIdx]) + { + sharedNeiLst->push_back(static_cast(featureId)); + sharedSAL->push_back(static_cast(surfaceArea)); + } + neighborList.setList(static_cast(featureIdx), sharedNeiLst); + sharedSurfaceAreaList.setList(static_cast(featureIdx), sharedSAL); + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp new file mode 100644 index 0000000000..1cc17c46c3 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp @@ -0,0 +1,51 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct ComputeFeatureNeighborsInputValues; + +/** + * @class ComputeFeatureNeighborsScanline + * @brief Out-of-core algorithm for ComputeFeatureNeighbors using Z-slice bulk I/O + * with per-face surface area accumulation. + * + * Reads FeatureIds one Z-slice at a time via copyIntoBuffer using a 3-slice rolling + * window (prev/cur/next) to resolve all 6 face neighbors with sequential disk access. + * BoundaryCells output is written per-slice via copyFromBuffer. + * + * Uses map-based per-feature surface area accumulation with per-face area values, + * matching Nathan Young's bug fix for correct surface area computation across + * faces of different sizes. + * + * Selected by DispatchAlgorithm when any input array is backed by ZarrStore. + * + * @see ComputeFeatureNeighborsDirect for the in-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + */ +class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsScanline +{ +public: + ComputeFeatureNeighborsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeFeatureNeighborsInputValues* inputValues); + ~ComputeFeatureNeighborsScanline() noexcept; + + ComputeFeatureNeighborsScanline(const ComputeFeatureNeighborsScanline&) = delete; + ComputeFeatureNeighborsScanline(ComputeFeatureNeighborsScanline&&) noexcept = delete; + ComputeFeatureNeighborsScanline& operator=(const ComputeFeatureNeighborsScanline&) = delete; + ComputeFeatureNeighborsScanline& operator=(ComputeFeatureNeighborsScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp index e160410194..d38426c59d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp @@ -8,12 +8,16 @@ #include "simplnx/Utilities/DataArrayUtilities.hpp" #include "simplnx/Utilities/MessageHelper.hpp" +#include + #include +#include using namespace nx::core; namespace { +constexpr usize k_ChunkTuples = 65536; constexpr int32 k_BadFeatureCount = -78231; constexpr uint64 k_MaxVoxelCount = std::numeric_limits::max(); /** @@ -37,17 +41,23 @@ Result<> ProcessImageGeom(ImageGeom& imageGeom, Float32AbstractDataStore& volume std::vector featureVoxelCounts(numFeatures, 0); msgHelper.sendMessage("Finding Voxel Counts..."); - // Count and store the number of voxels in each feature - for(usize voxelIdx = 0; voxelIdx < numVoxels; voxelIdx++) + // Count voxels per feature using chunked bulk I/O + auto featureIdBuf = std::make_unique(k_ChunkTuples); + for(usize offset = 0; offset < numVoxels; offset += k_ChunkTuples) { if(shouldCancel) { return {}; } - throttledMessenger.sendThrottledMessage([&] { return fmt::format(" - Counting || {:.2f}% Complete", CalculatePercentComplete(voxelIdx, numVoxels)); }); + throttledMessenger.sendThrottledMessage([&] { return fmt::format(" - Counting || {:.2f}% Complete", CalculatePercentComplete(offset, numVoxels)); }); - featureVoxelCounts[featureIds.getValue(voxelIdx)]++; + const usize count = std::min(k_ChunkTuples, numVoxels - offset); + featureIds.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), count)); + for(usize i = 0; i < count; i++) + { + featureVoxelCounts[featureIdBuf[i]]++; + } } const FloatVec3 spacing = imageGeom.getSpacing(); @@ -202,32 +212,32 @@ Result<> ProcessRectGridGeom(RectGridGeom& rectGridGeom, Float32AbstractDataStor std::vector featureCompensators(numFeatures, 0.0); msgHelper.sendMessage("Cell Level: Finding Voxel Counts and Summing Volumes..."); - // Count and store the number of voxels in each feature - for(usize voxelIdx = 0; voxelIdx < numVoxels; voxelIdx++) + // Count voxels and sum volumes using chunked bulk I/O + auto featureIdBuf = std::make_unique(k_ChunkTuples); + auto elemSizeBuf = std::make_unique(k_ChunkTuples); + for(usize offset = 0; offset < numVoxels; offset += k_ChunkTuples) { if(shouldCancel) { return {}; } - throttledMessenger.sendThrottledMessage([&] { return fmt::format(" - Calculating || {:.2f}% Complete", CalculatePercentComplete(voxelIdx, numVoxels)); }); - - const int32 voxelFeatureId = featureIds.getValue(voxelIdx); - featureVoxelCounts[voxelFeatureId]++; - - // Use Kahan summation to determine overall volume + throttledMessenger.sendThrottledMessage([&] { return fmt::format(" - Calculating || {:.2f}% Complete", CalculatePercentComplete(offset, numVoxels)); }); - // Attempt to recover low order into the value. The first instance is 0 - float64 value = static_cast(elemSizes.getValue(voxelIdx)) - featureCompensators[voxelFeatureId]; - - // low order may be lost - float64 volSum = featureVolumes[voxelFeatureId] + value; - - // recover and cache low order - featureCompensators[voxelFeatureId] = (volSum - featureVolumes[voxelFeatureId]) - value; - - // store volumes - featureVolumes[voxelFeatureId] = volSum; + const usize count = std::min(k_ChunkTuples, numVoxels - offset); + featureIds.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), count)); + elemSizes.copyIntoBuffer(offset, nonstd::span(elemSizeBuf.get(), count)); + for(usize i = 0; i < count; i++) + { + const int32 voxelFeatureId = featureIdBuf[i]; + featureVoxelCounts[voxelFeatureId]++; + + // Use Kahan summation to determine overall volume + float64 value = static_cast(elemSizeBuf[i]) - featureCompensators[voxelFeatureId]; + float64 volSum = featureVolumes[voxelFeatureId] + value; + featureCompensators[voxelFeatureId] = (volSum - featureVolumes[voxelFeatureId]) - value; + featureVolumes[voxelFeatureId] = volSum; + } } msgHelper.sendMessage("Feature Level: Storing Voxel Counts and Calculating ESD..."); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp new file mode 100644 index 0000000000..7bcc1844f8 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp @@ -0,0 +1,41 @@ +#include "ComputeKMedoids.hpp" + +#include "ComputeKMedoidsDirect.hpp" +#include "ComputeKMedoidsScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeKMedoids::ComputeKMedoids(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeKMedoids::~ComputeKMedoids() noexcept = default; + +// ----------------------------------------------------------------------------- +void ComputeKMedoids::updateProgress(const std::string& message) +{ + m_MessageHandler(IFilter::Message::Type::Info, message); +} + +// ----------------------------------------------------------------------------- +const std::atomic_bool& ComputeKMedoids::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> ComputeKMedoids::operator()() +{ + auto* clusteringArray = m_DataStructure.getDataAs(m_InputValues->ClusteringArrayPath); + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + return DispatchAlgorithm({clusteringArray, featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp new file mode 100644 index 0000000000..745b660209 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp @@ -0,0 +1,49 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ChoicesParameter.hpp" +#include "simplnx/Utilities/ClusteringUtilities.hpp" + +namespace nx::core +{ +struct SIMPLNXCORE_EXPORT KMedoidsInputValues +{ + uint64 InitClusters; + ClusterUtilities::DistanceMetric DistanceMetric; + DataPath ClusteringArrayPath; + DataPath MaskArrayPath; + DataPath FeatureIdsArrayPath; + DataPath MedoidsArrayPath; + uint64 Seed; +}; + +/** + * @class + */ +class SIMPLNXCORE_EXPORT ComputeKMedoids +{ +public: + ComputeKMedoids(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues); + ~ComputeKMedoids() noexcept; + + ComputeKMedoids(const ComputeKMedoids&) = delete; + ComputeKMedoids(ComputeKMedoids&&) noexcept = delete; + ComputeKMedoids& operator=(const ComputeKMedoids&) = delete; + ComputeKMedoids& operator=(ComputeKMedoids&&) noexcept = delete; + + Result<> operator()(); + void updateProgress(const std::string& message); + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const KMedoidsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp index 4676e42c31..0c5bae22f8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp @@ -1,10 +1,14 @@ #include "ComputeKMedoidsDirect.hpp" +#include "ComputeKMedoids.hpp" + #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/Utilities/ClusteringUtilities.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/MaskCompareUtilities.hpp" +#include + #include using namespace nx::core; @@ -73,6 +77,11 @@ class KMedoidsTemplate while(update) { + if(m_Filter->getCancel()) + { + return; + } + findClusters(numTuples, numCompDims); optClusterIdxs = clusterIdxs; @@ -94,7 +103,7 @@ class KMedoidsTemplate const AbstractDataStoreT& m_InputArray; AbstractDataStoreT& m_Medoids; const std::unique_ptr& m_Mask; - usize m_NumClusters; + usize m_NumClusters = 0; Int32AbstractDataStore& m_FeatureIds; ClusterUtilities::DistanceMetric m_DistMetric; std::mt19937_64::result_type m_Seed; @@ -182,7 +191,7 @@ class KMedoidsTemplate } // namespace // ----------------------------------------------------------------------------- -ComputeKMedoidsDirect::ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues) +ComputeKMedoidsDirect::ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const KMedoidsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -215,11 +224,10 @@ Result<> ComputeKMedoidsDirect::operator()() maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); } catch(const std::out_of_range& exception) { - // This really should NOT be happening as the path was verified during preflight BUT we may be calling this from - // somewhere else that is NOT going through the normal nx::core::IFilter API of Preflight and Execute std::string message = fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString()); return MakeErrorResult(-54070, message); } + RunTemplateClass(clusteringArray->getDataType(), this, clusteringArray, m_DataStructure.getDataAs(m_InputValues->MedoidsArrayPath), maskCompare, m_InputValues->InitClusters, m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(), m_InputValues->DistanceMetric, m_InputValues->Seed); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp index 8f51562a63..d4cbba8441 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp @@ -2,32 +2,23 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ChoicesParameter.hpp" -#include "simplnx/Utilities/ClusteringUtilities.hpp" namespace nx::core { -struct SIMPLNXCORE_EXPORT KMedoidsInputValues -{ - uint64 InitClusters; - ClusterUtilities::DistanceMetric DistanceMetric; - DataPath ClusteringArrayPath; - DataPath MaskArrayPath; - DataPath FeatureIdsArrayPath; - DataPath MedoidsArrayPath; - uint64 Seed; -}; +struct KMedoidsInputValues; /** - * @class + * @class ComputeKMedoidsDirect + * @brief In-core algorithm for ComputeKMedoids. Uses direct per-element operator[] + * access for distance computation, cluster assignment, and medoid optimization. + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. */ class SIMPLNXCORE_EXPORT ComputeKMedoidsDirect { public: - ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues); + ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const KMedoidsInputValues* inputValues); ~ComputeKMedoidsDirect() noexcept; ComputeKMedoidsDirect(const ComputeKMedoidsDirect&) = delete; @@ -36,6 +27,7 @@ class SIMPLNXCORE_EXPORT ComputeKMedoidsDirect ComputeKMedoidsDirect& operator=(ComputeKMedoidsDirect&&) noexcept = delete; Result<> operator()(); + void updateProgress(const std::string& message); const std::atomic_bool& getCancel(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp new file mode 100644 index 0000000000..f9b0b1004b --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp @@ -0,0 +1,277 @@ +#include "ComputeKMedoidsScanline.hpp" + +#include "ComputeKMedoids.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/ClusteringUtilities.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" +#include "simplnx/Utilities/MaskCompareUtilities.hpp" + +#include + +#include + +using namespace nx::core; + +namespace +{ +template +class KMedoidsTemplate +{ +public: + KMedoidsTemplate(ComputeKMedoidsScanline* filter, const IDataArray* inputIDataArray, IDataArray* medoidsIDataArray, const std::unique_ptr& maskDataArray, + usize numClusters, Int32AbstractDataStore& fIds, ClusterUtilities::DistanceMetric distMetric, std::mt19937_64::result_type seed) + : m_Filter(filter) + , m_InputArray(inputIDataArray->template getIDataStoreRefAs>()) + , m_Medoids(medoidsIDataArray->template getIDataStoreRefAs>()) + , m_Mask(maskDataArray) + , m_NumClusters(numClusters) + , m_FeatureIds(fIds) + , m_DistMetric(distMetric) + , m_Seed(seed) + { + } + ~KMedoidsTemplate() = default; + + KMedoidsTemplate(const KMedoidsTemplate&) = delete; // Copy Constructor Not Implemented + void operator=(const KMedoidsTemplate&) = delete; // Move assignment Not Implemented + + // ----------------------------------------------------------------------------- + void operator()() + { + usize numTuples = m_InputArray.getNumberOfTuples(); + int32 numCompDims = m_InputArray.getNumberOfComponents(); + + std::mt19937_64 gen(m_Seed); + std::uniform_int_distribution dist(0, numTuples - 1); + + std::vector clusterIdxs(m_NumClusters); + + usize clusterChoices = 0; + while(clusterChoices < m_NumClusters) + { + usize index = dist(gen); + if(m_Mask->isTrue(index)) + { + clusterIdxs[clusterChoices] = index; + clusterChoices++; + } + } + + // OOC: use bulk I/O to initialize medoids + auto tupleBuf = std::make_unique(numCompDims); + for(usize i = 0; i < m_NumClusters; i++) + { + m_InputArray.copyIntoBuffer(numCompDims * clusterIdxs[i], nonstd::span(tupleBuf.get(), numCompDims)); + m_Medoids.copyFromBuffer(numCompDims * (i + 1), nonstd::span(tupleBuf.get(), numCompDims)); + } + + findClusters(numTuples, numCompDims); + + std::vector optClusterIdxs(clusterIdxs); + + std::vector costs = optimizeClusters(numTuples, numCompDims, clusterIdxs); + + bool update = optClusterIdxs == clusterIdxs ? false : true; + usize iteration = 1; + + while(update) + { + if(m_Filter->getCancel()) + { + return; + } + + findClusters(numTuples, numCompDims); + + optClusterIdxs = clusterIdxs; + + costs = optimizeClusters(numTuples, numCompDims, clusterIdxs); + + update = optClusterIdxs == clusterIdxs ? false : true; + + float64 sum = std::accumulate(std::begin(costs), std::end(costs), 0.0); + m_Filter->updateProgress(fmt::format("Clustering Data || Iteration {} || Total Cost: {}", iteration, sum)); + iteration++; + } + } + +private: + using DataArrayT = DataArray; + using AbstractDataStoreT = AbstractDataStore; + ComputeKMedoidsScanline* m_Filter; + const AbstractDataStoreT& m_InputArray; + AbstractDataStoreT& m_Medoids; + const std::unique_ptr& m_Mask; + usize m_NumClusters = 0; + Int32AbstractDataStore& m_FeatureIds; + ClusterUtilities::DistanceMetric m_DistMetric; + std::mt19937_64::result_type m_Seed; + + // ----------------------------------------------------------------------------- + // OOC: cache medoids locally, process input and featureIds in chunks + void findClusters(usize tuples, int32 dims) + { + // Cache medoids (small: numClusters * dims) + const usize medoidsSize = (m_NumClusters + 1) * dims; + std::vector medoidsCache(medoidsSize); + m_Medoids.copyIntoBuffer(0, nonstd::span(medoidsCache.data(), medoidsSize)); + + constexpr usize k_ChunkTuples = 65536; + auto inputBuf = std::make_unique(k_ChunkTuples * dims); + std::vector fidsBuf(k_ChunkTuples); + + for(usize startTup = 0; startTup < tuples; startTup += k_ChunkTuples) + { + if(m_Filter->getCancel()) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, tuples); + const usize count = endTup - startTup; + + m_InputArray.copyIntoBuffer(startTup * dims, nonstd::span(inputBuf.get(), count * dims)); + m_FeatureIds.copyIntoBuffer(startTup, nonstd::span(fidsBuf.data(), count)); + + for(usize local = 0; local < count; local++) + { + if(m_Mask->isTrue(startTup + local)) + { + float64 minDist = std::numeric_limits::max(); + for(int32 j = 0; j < m_NumClusters; j++) + { + float64 dist = ClusterUtilities::GetDistance(inputBuf.get(), (dims * local), medoidsCache, (dims * (j + 1)), dims, m_DistMetric); + if(dist < minDist) + { + minDist = dist; + fidsBuf[local] = j + 1; + } + } + } + } + + m_FeatureIds.copyFromBuffer(startTup, nonstd::span(fidsBuf.data(), count)); + } + } + + // ----------------------------------------------------------------------------- + // OOC: process one cluster at a time to avoid O(n) total member list allocation. + // Peak memory is O(max_cluster_size), not O(n). + std::vector optimizeClusters(usize tuples, int32 dims, std::vector& clusterIdxs) + { + std::vector minCosts(m_NumClusters, std::numeric_limits::max()); + + constexpr usize k_ChunkSize = 65536; + std::vector fidsBuf(k_ChunkSize); + auto tupleBufJ = std::make_unique(dims); + auto tupleBufK = std::make_unique(dims); + + // Process one cluster at a time — build member list, compute costs, then release + for(usize i = 0; i < m_NumClusters; i++) + { + if(m_Filter->getCancel()) + { + return {}; + } + // Scan featureIds in chunks to find members of cluster i only + std::vector members; + for(usize start = 0; start < tuples; start += k_ChunkSize) + { + usize count = std::min(k_ChunkSize, tuples - start); + m_FeatureIds.copyIntoBuffer(start, nonstd::span(fidsBuf.data(), count)); + for(usize local = 0; local < count; local++) + { + if(fidsBuf[local] == static_cast(i + 1) && m_Mask->isTrue(start + local)) + { + members.push_back(start + local); + } + } + } + + // Find the member that minimizes total intra-cluster distance + for(usize mj = 0; mj < members.size(); mj++) + { + if(m_Filter->getCancel()) + { + return {}; + } + usize j = members[mj]; + m_InputArray.copyIntoBuffer(j * dims, nonstd::span(tupleBufJ.get(), dims)); + + float64 cost = 0.0; + for(usize mk = 0; mk < members.size(); mk++) + { + if(m_Filter->getCancel()) + { + return {}; + } + usize k = members[mk]; + m_InputArray.copyIntoBuffer(k * dims, nonstd::span(tupleBufK.get(), dims)); + cost += ClusterUtilities::GetDistance(tupleBufJ.get(), 0, tupleBufK.get(), 0, dims, m_DistMetric); + } + + if(cost < minCosts[i]) + { + minCosts[i] = cost; + clusterIdxs[i] = j; + } + } + // members is released here at end of loop iteration + } + + // Update medoids from best candidates + for(usize i = 0; i < m_NumClusters; i++) + { + m_InputArray.copyIntoBuffer(dims * clusterIdxs[i], nonstd::span(tupleBufJ.get(), dims)); + m_Medoids.copyFromBuffer(dims * (i + 1), nonstd::span(tupleBufJ.get(), dims)); + } + + return minCosts; + } +}; +} // namespace + +// ----------------------------------------------------------------------------- +ComputeKMedoidsScanline::ComputeKMedoidsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const KMedoidsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeKMedoidsScanline::~ComputeKMedoidsScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +void ComputeKMedoidsScanline::updateProgress(const std::string& message) +{ + m_MessageHandler(IFilter::Message::Type::Info, message); +} + +// ----------------------------------------------------------------------------- +const std::atomic_bool& ComputeKMedoidsScanline::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> ComputeKMedoidsScanline::operator()() +{ + auto* clusteringArray = m_DataStructure.getDataAs(m_InputValues->ClusteringArrayPath); + std::unique_ptr maskCompare; + try + { + maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); + } catch(const std::out_of_range& exception) + { + std::string message = fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString()); + return MakeErrorResult(-54070, message); + } + + RunTemplateClass(clusteringArray->getDataType(), this, clusteringArray, m_DataStructure.getDataAs(m_InputValues->MedoidsArrayPath), maskCompare, + m_InputValues->InitClusters, m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(), + m_InputValues->DistanceMetric, m_InputValues->Seed); + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp new file mode 100644 index 0000000000..e2af49976e --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp @@ -0,0 +1,41 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct KMedoidsInputValues; + +/** + * @class ComputeKMedoidsScanline + * @brief Out-of-core algorithm for ComputeKMedoids. Uses chunked copyIntoBuffer/copyFromBuffer + * bulk I/O for distance computation, cluster assignment, and medoid optimization. + * Selected by DispatchAlgorithm when any input array is backed by out-of-core storage. + */ +class SIMPLNXCORE_EXPORT ComputeKMedoidsScanline +{ +public: + ComputeKMedoidsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const KMedoidsInputValues* inputValues); + ~ComputeKMedoidsScanline() noexcept; + + ComputeKMedoidsScanline(const ComputeKMedoidsScanline&) = delete; + ComputeKMedoidsScanline(ComputeKMedoidsScanline&&) noexcept = delete; + ComputeKMedoidsScanline& operator=(const ComputeKMedoidsScanline&) = delete; + ComputeKMedoidsScanline& operator=(ComputeKMedoidsScanline&&) noexcept = delete; + + Result<> operator()(); + + void updateProgress(const std::string& message); + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const KMedoidsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp new file mode 100644 index 0000000000..155fb3188a --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp @@ -0,0 +1,35 @@ +#include "ComputeSurfaceAreaToVolume.hpp" + +#include "ComputeSurfaceAreaToVolumeDirect.hpp" +#include "ComputeSurfaceAreaToVolumeScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeSurfaceAreaToVolume::ComputeSurfaceAreaToVolume(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeSurfaceAreaToVolumeInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeSurfaceAreaToVolume::~ComputeSurfaceAreaToVolume() noexcept = default; + +// ----------------------------------------------------------------------------- +const std::atomic_bool& ComputeSurfaceAreaToVolume::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> ComputeSurfaceAreaToVolume::operator()() +{ + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp new file mode 100644 index 0000000000..637caf9980 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp @@ -0,0 +1,47 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeInputValues +{ + DataPath FeatureIdsArrayPath; + DataPath NumCellsArrayPath; + DataPath SurfaceAreaVolumeRatioArrayName; + bool CalculateSphericity; + DataPath SphericityArrayName; + DataPath InputImageGeometry; +}; + +/** + * @class + */ +class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolume +{ +public: + ComputeSurfaceAreaToVolume(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues); + ~ComputeSurfaceAreaToVolume() noexcept; + + ComputeSurfaceAreaToVolume(const ComputeSurfaceAreaToVolume&) = delete; + ComputeSurfaceAreaToVolume(ComputeSurfaceAreaToVolume&&) noexcept = delete; + ComputeSurfaceAreaToVolume& operator=(const ComputeSurfaceAreaToVolume&) = delete; + ComputeSurfaceAreaToVolume& operator=(ComputeSurfaceAreaToVolume&&) noexcept = delete; + + Result<> operator()(); + + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp index f08c4de596..923e17b67b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp @@ -1,16 +1,20 @@ #include "ComputeSurfaceAreaToVolumeDirect.hpp" +#include "ComputeSurfaceAreaToVolume.hpp" + #include "simplnx/Common/Constants.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataGroup.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" +#include + using namespace nx::core; // ----------------------------------------------------------------------------- ComputeSurfaceAreaToVolumeDirect::ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - ComputeSurfaceAreaToVolumeInputValues* inputValues) + const ComputeSurfaceAreaToVolumeInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -22,12 +26,11 @@ ComputeSurfaceAreaToVolumeDirect::ComputeSurfaceAreaToVolumeDirect(DataStructure ComputeSurfaceAreaToVolumeDirect::~ComputeSurfaceAreaToVolumeDirect() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ComputeSurfaceAreaToVolumeDirect::getCancel() -{ - return m_ShouldCancel; -} - -// ----------------------------------------------------------------------------- +/** + * @brief Computes surface-area-to-volume ratio using direct Z-Y-X iteration. + * In-core path: accumulates per-feature surface area from face-neighbor + * comparisons, then divides by voxel volume. Optionally computes sphericity. + */ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() { // Input Cell Data @@ -52,16 +55,14 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() SizeVec3 dims = imageGeom.getDimensions(); FloatVec3 spacing = imageGeom.getSpacing(); - auto xPoints = static_cast(dims[0]); - auto yPoints = static_cast(dims[1]); - auto zPoints = static_cast(dims[2]); + auto xPoints = static_cast(dims[0]); + auto yPoints = static_cast(dims[1]); + auto zPoints = static_cast(dims[2]); float32 voxelVol = spacing[0] * spacing[1] * spacing[2]; - std::vector featureSurfaceArea(static_cast(numFeatures), 0.0f); + std::vector featureSurfaceArea(static_cast(numFeatures), 0.0f); - // This stores an offset to get to a particular index in the array based on - // a normal orthogonal cube int64 neighborOffset[6] = {0, 0, 0, 0, 0, 0}; neighborOffset[0] = -xPoints * yPoints; // -Z neighborOffset[1] = -xPoints; // -Y @@ -70,57 +71,52 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() neighborOffset[4] = xPoints; // +Y neighborOffset[5] = xPoints * yPoints; // +Z - // Start looping over the regular grid data (This could be either an Image Geometry or a Rectilinear Grid geometry (in theory) for(int64 zIdx = 0; zIdx < zPoints; zIdx++) { - m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Computing Z Slice: '{}'", zIdx)); if(m_ShouldCancel) { return {}; } - int64 zStride = zIdx * xPoints * yPoints; for(int64 yIdx = 0; yIdx < yPoints; yIdx++) { int64 yStride = yIdx * xPoints; for(int64 xIdx = 0; xIdx < xPoints; xIdx++) { - float onSurface = 0.0f; // Start totalling the surface area + float32 onSurface = 0.0f; int32 currentFeatureId = featureIdsStoreRef[zStride + yStride + xIdx]; - // If the current feature ID is not valid (< 1), then just continue; if(currentFeatureId < 1) { continue; } - // Loop over all 6 face neighbors for(int32 neighborOffsetIndex = 0; neighborOffsetIndex < 6; neighborOffsetIndex++) { - if(neighborOffsetIndex == 0 && zIdx == 0) // if we are on the bottom Z Layer, skip + if(neighborOffsetIndex == 0 && zIdx == 0) { continue; } - if(neighborOffsetIndex == 5 && zIdx == (zPoints - 1)) // if we are on the top Z Layer, skip + if(neighborOffsetIndex == 5 && zIdx == (zPoints - 1)) { continue; } - if(neighborOffsetIndex == 1 && yIdx == 0) // If we are on the first Y row, skip + if(neighborOffsetIndex == 1 && yIdx == 0) { continue; } - if(neighborOffsetIndex == 4 && yIdx == (yPoints - 1)) // If we are on the last Y row, skip + if(neighborOffsetIndex == 4 && yIdx == (yPoints - 1)) { continue; } - if(neighborOffsetIndex == 2 && xIdx == 0) // If we are on the first X column, skip + if(neighborOffsetIndex == 2 && xIdx == 0) { continue; } - if(neighborOffsetIndex == 3 && xIdx == (xPoints - 1)) // If we are on the last X column, skip + if(neighborOffsetIndex == 3 && xIdx == (xPoints - 1)) { continue; } - // + int64 neighborIndex = zStride + yStride + xIdx + neighborOffset[neighborOffsetIndex]; if(featureIdsStoreRef[neighborIndex] != currentFeatureId) @@ -148,18 +144,18 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() const float32 thirdRootPi = std::pow(nx::core::Constants::k_PiF, 0.333333f); for(usize i = 1; i < numFeatures; i++) { - float featureVolume = voxelVol * numCells[i]; + float32 featureVolume = voxelVol * numCells[i]; surfaceAreaVolumeRatio[i] = featureSurfaceArea[i] / featureVolume; } - if(m_InputValues->CalculateSphericity) // Calc the sphericity if requested + if(m_InputValues->CalculateSphericity) { m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Computing Sphericity")); auto& sphericity = m_DataStructure.getDataAs(m_InputValues->SphericityArrayName)->getDataStoreRef(); for(usize i = 1; i < static_cast(numFeatures); i++) { - float featureVolume = voxelVol * numCells[i]; + float32 featureVolume = voxelVol * numCells[i]; sphericity[i] = (thirdRootPi * std::pow((6.0f * featureVolume), 0.66666f)) / featureSurfaceArea[i]; } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp index 560ef2c1c6..54e3160442 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp @@ -2,30 +2,24 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" namespace nx::core { - -struct SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeInputValues -{ - DataPath FeatureIdsArrayPath; - DataPath NumCellsArrayPath; - DataPath SurfaceAreaVolumeRatioArrayName; - bool CalculateSphericity; - DataPath SphericityArrayName; - DataPath InputImageGeometry; -}; +struct ComputeSurfaceAreaToVolumeInputValues; /** - * @class + * @class ComputeSurfaceAreaToVolumeDirect + * @brief In-core algorithm for ComputeSurfaceAreaToVolume. Preserves the original sequential + * Z-Y-X voxel iteration with face-neighbor surface area accumulation per feature. + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. */ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeDirect { public: - ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues); + ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeSurfaceAreaToVolumeInputValues* inputValues); ~ComputeSurfaceAreaToVolumeDirect() noexcept; ComputeSurfaceAreaToVolumeDirect(const ComputeSurfaceAreaToVolumeDirect&) = delete; @@ -35,8 +29,6 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeDirect Result<> operator()(); - const std::atomic_bool& getCancel(); - private: DataStructure& m_DataStructure; const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp new file mode 100644 index 0000000000..f53f25c0bb --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp @@ -0,0 +1,203 @@ +#include "ComputeSurfaceAreaToVolumeScanline.hpp" + +#include "ComputeSurfaceAreaToVolume.hpp" + +#include "simplnx/Common/Constants.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/DataArrayUtilities.hpp" + +#include +#include + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeSurfaceAreaToVolumeScanline::ComputeSurfaceAreaToVolumeScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeSurfaceAreaToVolumeInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeSurfaceAreaToVolumeScanline::~ComputeSurfaceAreaToVolumeScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief Computes surface-area-to-volume ratio using Z-slice bulk I/O with a rolling window. + * OOC path: reads featureIds one Z-slice at a time via copyIntoBuffer(), + * keeping 3 slices resident (prev/cur/next) for cross-slice neighbour access. + * Same logic as ComputeSurfaceAreaToVolumeDirect. + */ +Result<> ComputeSurfaceAreaToVolumeScanline::operator()() +{ + // Input Cell Data + auto featureIdsArrayPtr = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + auto& featureIdsStore = featureIdsArrayPtr->getDataStoreRef(); + + // Input Feature Data + const auto& numCells = m_DataStructure.getDataAs(m_InputValues->NumCellsArrayPath)->getDataStoreRef(); + + // Output Feature Data + auto& surfaceAreaVolumeRatio = m_DataStructure.getDataAs(m_InputValues->SurfaceAreaVolumeRatioArrayName)->getDataStoreRef(); + + // Required Geometry + const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometry); + + auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, m_InputValues->NumCellsArrayPath.getParent(), *featureIdsArrayPtr, false, m_MessageHandler); + if(validateNumFeatResult.invalid()) + { + return validateNumFeatResult; + } + auto numFeatures = static_cast(numCells.getNumberOfTuples()); + SizeVec3 dims = imageGeom.getDimensions(); + FloatVec3 spacing = imageGeom.getSpacing(); + + auto xPoints = static_cast(dims[0]); + auto yPoints = static_cast(dims[1]); + auto zPoints = static_cast(dims[2]); + + float32 voxelVol = spacing[0] * spacing[1] * spacing[2]; + + std::vector featureSurfaceArea(static_cast(numFeatures), 0.0f); + + // Face areas for each neighbor direction + const float32 xyFaceArea = spacing[0] * spacing[1]; // XY face shared (Z-normal) + const float32 yzFaceArea = spacing[1] * spacing[2]; // YZ face shared (X-normal) + const float32 zxFaceArea = spacing[2] * spacing[0]; // XZ face shared (Y-normal) + + // Z-slice rolling window for bulk I/O + const usize sliceSize = static_cast(yPoints) * static_cast(xPoints); + std::vector prevSlice(sliceSize, 0); + std::vector curSlice(sliceSize, 0); + std::vector nextSlice(sliceSize, 0); + + // Load initial slices + featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + if(zPoints > 1) + { + featureIdsStore.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + + for(int64 z = 0; z < zPoints; z++) + { + if(m_ShouldCancel) + { + return {}; + } + for(int64 y = 0; y < yPoints; y++) + { + for(int64 x = 0; x < xPoints; x++) + { + const usize inSlice = static_cast(y) * static_cast(xPoints) + static_cast(x); + int32 currentFeatureId = curSlice[inSlice]; + if(currentFeatureId < 1) + { + continue; + } + + float32 onSurface = 0.0f; + + // -Z neighbor (index 0): check prevSlice at same (y,x) + if(z > 0) + { + if(prevSlice[inSlice] != currentFeatureId) + { + onSurface += xyFaceArea; + } + } + + // +Z neighbor (index 5): check nextSlice at same (y,x) + if(z < zPoints - 1) + { + if(nextSlice[inSlice] != currentFeatureId) + { + onSurface += xyFaceArea; + } + } + + // -Y neighbor (index 1): check curSlice at (y-1, x) + if(y > 0) + { + if(curSlice[inSlice - static_cast(xPoints)] != currentFeatureId) + { + onSurface += yzFaceArea; + } + } + + // +Y neighbor (index 4): check curSlice at (y+1, x) + if(y < yPoints - 1) + { + if(curSlice[inSlice + static_cast(xPoints)] != currentFeatureId) + { + onSurface += yzFaceArea; + } + } + + // -X neighbor (index 2): check curSlice at (y, x-1) + if(x > 0) + { + if(curSlice[inSlice - 1] != currentFeatureId) + { + onSurface += zxFaceArea; + } + } + + // +X neighbor (index 3): check curSlice at (y, x+1) + if(x < xPoints - 1) + { + if(curSlice[inSlice + 1] != currentFeatureId) + { + onSurface += zxFaceArea; + } + } + + featureSurfaceArea[currentFeatureId] += onSurface; + } + } + + // Shift the rolling window + std::swap(prevSlice, curSlice); + std::swap(curSlice, nextSlice); + if(z + 2 < zPoints) + { + featureIdsStore.copyIntoBuffer(static_cast(z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + } + + // Cache the feature-level numCells array locally to avoid per-element OOC lookups + const usize numFeaturesUSize = static_cast(numFeatures); + std::vector localNumCells(numFeaturesUSize); + numCells.copyIntoBuffer(0, nonstd::span(localNumCells.data(), numFeaturesUSize)); + + // Compute results into a local buffer, then bulk-write to OOC store + std::vector localSurfaceAreaVolumeRatio(numFeaturesUSize, 0.0f); + + const float32 thirdRootPi = std::pow(nx::core::Constants::k_PiF, 0.333333f); + for(usize i = 1; i < numFeaturesUSize; i++) + { + float32 featureVolume = voxelVol * localNumCells[i]; + localSurfaceAreaVolumeRatio[i] = featureSurfaceArea[i] / featureVolume; + } + surfaceAreaVolumeRatio.copyFromBuffer(0, nonstd::span(localSurfaceAreaVolumeRatio.data(), numFeaturesUSize)); + + if(m_InputValues->CalculateSphericity) + { + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Computing Sphericity")); + + auto& sphericity = m_DataStructure.getDataAs(m_InputValues->SphericityArrayName)->getDataStoreRef(); + std::vector localSphericity(numFeaturesUSize, 0.0f); + for(usize i = 1; i < numFeaturesUSize; i++) + { + float32 featureVolume = voxelVol * localNumCells[i]; + localSphericity[i] = (thirdRootPi * std::pow((6.0f * featureVolume), 0.66666f)) / featureSurfaceArea[i]; + } + sphericity.copyFromBuffer(0, nonstd::span(localSphericity.data(), numFeaturesUSize)); + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp new file mode 100644 index 0000000000..20b14a2bd8 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp @@ -0,0 +1,40 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct ComputeSurfaceAreaToVolumeInputValues; + +/** + * @class ComputeSurfaceAreaToVolumeScanline + * @brief Out-of-core algorithm for ComputeSurfaceAreaToVolume. Wraps the voxel iteration in + * chunk-sequential access for guaranteed sequential disk I/O on ZarrStore-backed arrays. + * Feature-level ratio and sphericity computations are unchanged. Selected by DispatchAlgorithm + * when any input array is backed by ZarrStore. + */ +class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeScanline +{ +public: + ComputeSurfaceAreaToVolumeScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeSurfaceAreaToVolumeInputValues* inputValues); + ~ComputeSurfaceAreaToVolumeScanline() noexcept; + + ComputeSurfaceAreaToVolumeScanline(const ComputeSurfaceAreaToVolumeScanline&) = delete; + ComputeSurfaceAreaToVolumeScanline(ComputeSurfaceAreaToVolumeScanline&&) noexcept = delete; + ComputeSurfaceAreaToVolumeScanline& operator=(const ComputeSurfaceAreaToVolumeScanline&) = delete; + ComputeSurfaceAreaToVolumeScanline& operator=(ComputeSurfaceAreaToVolumeScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp new file mode 100644 index 0000000000..6997eaeafb --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp @@ -0,0 +1,29 @@ +#include "ComputeSurfaceFeatures.hpp" + +#include "ComputeSurfaceFeaturesDirect.hpp" +#include "ComputeSurfaceFeaturesScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +ComputeSurfaceFeatures::ComputeSurfaceFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + ComputeSurfaceFeaturesInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeSurfaceFeatures::~ComputeSurfaceFeatures() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> ComputeSurfaceFeatures::operator()() +{ + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath); + return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp new file mode 100644 index 0000000000..0c825a2b1b --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp @@ -0,0 +1,51 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ArraySelectionParameter.hpp" +#include "simplnx/Parameters/AttributeMatrixSelectionParameter.hpp" +#include "simplnx/Parameters/BoolParameter.hpp" +#include "simplnx/Parameters/DataObjectNameParameter.hpp" +#include "simplnx/Parameters/GeometrySelectionParameter.hpp" + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesInputValues +{ + AttributeMatrixSelectionParameter::ValueType FeatureAttributeMatrixPath; + ArraySelectionParameter::ValueType FeatureIdsPath; + GeometrySelectionParameter::ValueType InputImageGeometryPath; + BoolParameter::ValueType MarkFeature0Neighbors; + DataObjectNameParameter::ValueType SurfaceFeaturesArrayName; +}; + +/** + * @class ComputeSurfaceFeatures + * @brief This algorithm implements support code for the ComputeSurfaceFeaturesFilter + */ + +class SIMPLNXCORE_EXPORT ComputeSurfaceFeatures +{ +public: + ComputeSurfaceFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues); + ~ComputeSurfaceFeatures() noexcept; + + ComputeSurfaceFeatures(const ComputeSurfaceFeatures&) = delete; + ComputeSurfaceFeatures(ComputeSurfaceFeatures&&) noexcept = delete; + ComputeSurfaceFeatures& operator=(const ComputeSurfaceFeatures&) = delete; + ComputeSurfaceFeatures& operator=(ComputeSurfaceFeatures&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp index 82b9006d49..7dd9731578 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp @@ -1,5 +1,7 @@ #include "ComputeSurfaceFeaturesDirect.hpp" +#include "ComputeSurfaceFeatures.hpp" + #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" @@ -103,20 +105,20 @@ void findSurfaceFeatures3D(DataStructure& dataStructure, const DataPath& feature const usize xPoints = featureGeometry.getNumXCells(); const usize yPoints = featureGeometry.getNumYCells(); const usize zPoints = featureGeometry.getNumZCells(); + const usize totalSlices = zPoints; for(usize z = 0; z < zPoints; z++) { + if(shouldCancel) + { + return; + } const usize zStride = z * xPoints * yPoints; for(usize y = 0; y < yPoints; y++) { const usize yStride = y * xPoints; for(usize x = 0; x < xPoints; x++) { - if(shouldCancel) - { - return; - } - const int32 gNum = featureIds[zStride + yStride + x]; if(gNum != 0 && !surfaceFeatures[gNum]) { @@ -158,15 +160,14 @@ void findSurfaceFeatures2D(DataStructure& dataStructure, const DataPath& feature for(usize y = 0; y < yPoints; y++) { + if(shouldCancel) + { + return; + } const usize yStride = y * xPoints; for(usize x = 0; x < xPoints; x++) { - if(shouldCancel) - { - return; - } - const int32 gNum = featureIds[yStride + x]; if(gNum != 0 && surfaceFeatures[gNum] == 0) { @@ -182,7 +183,7 @@ void findSurfaceFeatures2D(DataStructure& dataStructure, const DataPath& feature // ----------------------------------------------------------------------------- ComputeSurfaceFeaturesDirect::ComputeSurfaceFeaturesDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - ComputeSurfaceFeaturesInputValues* inputValues) + const ComputeSurfaceFeaturesInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -194,16 +195,19 @@ ComputeSurfaceFeaturesDirect::ComputeSurfaceFeaturesDirect(DataStructure& dataSt ComputeSurfaceFeaturesDirect::~ComputeSurfaceFeaturesDirect() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Identifies surface features using direct Z-Y-X iteration. + * In-core path: delegates to findSurfaceFeatures3D or findSurfaceFeatures2D + * depending on geometry dimensionality. + */ Result<> ComputeSurfaceFeaturesDirect::operator()() { - const auto pMarkFeature0NeighborsValue = m_InputValues->MarkFeature0Neighbors; const auto pFeatureGeometryPathValue = m_InputValues->InputImageGeometryPath; const auto pFeatureIdsArrayPathValue = m_InputValues->FeatureIdsPath; const auto pFeaturesAttributeMatrixPathValue = m_InputValues->FeatureAttributeMatrixPath; const auto pSurfaceFeaturesArrayPathValue = pFeaturesAttributeMatrixPathValue.createChildPath(m_InputValues->SurfaceFeaturesArrayName); - // Resize the surface features array to the proper size const auto& featureIdsArray = m_DataStructure.getDataRefAs(pFeatureIdsArrayPathValue); auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, pFeaturesAttributeMatrixPathValue, featureIdsArray, false, m_MessageHandler); @@ -212,7 +216,6 @@ Result<> ComputeSurfaceFeaturesDirect::operator()() return validateNumFeatResult; } - // Find surface features const auto& featureGeometry = m_DataStructure.getDataRefAs(pFeatureGeometryPathValue); if(const usize geometryDimensionality = featureGeometry.getDimensionality(); geometryDimensionality == 3) { @@ -224,7 +227,7 @@ Result<> ComputeSurfaceFeaturesDirect::operator()() } else { - MakeErrorResult(-1000, fmt::format("Image Geometry at path '{}' must be either 3D or 2D", pFeatureGeometryPathValue.toString())); + return MakeErrorResult(-1000, fmt::format("Image Geometry at path '{}' must be either 3D or 2D", pFeatureGeometryPathValue.toString())); } return {}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp index 8be8b9087c..ebacffdb14 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp @@ -2,36 +2,23 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ArraySelectionParameter.hpp" -#include "simplnx/Parameters/AttributeMatrixSelectionParameter.hpp" -#include "simplnx/Parameters/BoolParameter.hpp" -#include "simplnx/Parameters/DataObjectNameParameter.hpp" -#include "simplnx/Parameters/GeometrySelectionParameter.hpp" namespace nx::core { - -struct SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesInputValues -{ - AttributeMatrixSelectionParameter::ValueType FeatureAttributeMatrixPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - BoolParameter::ValueType MarkFeature0Neighbors; - DataObjectNameParameter::ValueType SurfaceFeaturesArrayName; -}; +struct ComputeSurfaceFeaturesInputValues; /** * @class ComputeSurfaceFeaturesDirect - * @brief This algorithm implements support code for the ComputeSurfaceFeaturesFilter + * @brief In-core algorithm for ComputeSurfaceFeatures. Preserves the original 2D/3D branching + * with sequential voxel iteration and face-neighbor surface detection. Selected by + * DispatchAlgorithm when all input arrays are backed by in-memory DataStore. */ - class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesDirect { public: - ComputeSurfaceFeaturesDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues); + ComputeSurfaceFeaturesDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceFeaturesInputValues* inputValues); ~ComputeSurfaceFeaturesDirect() noexcept; ComputeSurfaceFeaturesDirect(const ComputeSurfaceFeaturesDirect&) = delete; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp new file mode 100644 index 0000000000..88a852252f --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp @@ -0,0 +1,316 @@ +#include "ComputeSurfaceFeaturesScanline.hpp" + +#include "ComputeSurfaceFeatures.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/DataArrayUtilities.hpp" + +#include + +#include + +using namespace nx::core; + +namespace +{ +/** + * @brief Checks whether a voxel at (remappedX, remappedY) in a 2D geometry is + * on the surface. Operates on a pair of flat buffers that hold the current + * remapped-Y row and its neighbours in the remapped-Y direction. + * + * For the degenerate-Z case (zPoints==1) the entire dataset is one Z-slice, + * so prevRow/nextRow are unused for Z neighbours -- all four 2D neighbours + * live in curSlice. + * + * For the degenerate-X or degenerate-Y cases, the remapped-Y direction maps + * to the native Z axis, so prevRow/nextRow come from prevSlice/nextSlice. + */ +bool IsPointASurfaceFeature2D(usize remappedX, usize remappedY, usize remappedXPoints, usize remappedYPoints, bool markFeature0Neighbors, const std::vector& curSlice, + const std::vector& prevSlice, const std::vector& nextSlice, usize nativeInSlice, bool hasPrevSlice, bool hasNextSlice, bool degenerateZ) +{ + if(remappedX <= 0 || remappedX >= remappedXPoints - 1) + { + return true; + } + if(remappedY <= 0 || remappedY >= remappedYPoints - 1) + { + return true; + } + + if(markFeature0Neighbors) + { + if(degenerateZ) + { + // All 4 neighbours are in curSlice (the single Z-slice holds everything) + // For degenerate Z: remapped coords map directly to native Y*xPoints+X layout + const usize yStride = remappedY * remappedXPoints; + if(curSlice[yStride + remappedX - 1] == 0) + { + return true; + } + if(curSlice[yStride + remappedX + 1] == 0) + { + return true; + } + if(curSlice[(remappedY - 1) * remappedXPoints + remappedX] == 0) + { + return true; + } + if(curSlice[(remappedY + 1) * remappedXPoints + remappedX] == 0) + { + return true; + } + } + else + { + // Remapped Y direction maps to native Z, so +/-Y neighbours come + // from prevSlice / nextSlice. Remapped X neighbours are within + // curSlice at stride ±1 from the native in-slice index (works for + // both degenerate-X and degenerate-Y because the non-degenerate + // in-plane dimension is always contiguous in memory). + + // -remappedX neighbour + if(curSlice[nativeInSlice - 1] == 0) + { + return true; + } + // +remappedX neighbour + if(curSlice[nativeInSlice + 1] == 0) + { + return true; + } + // -remappedY neighbour (previous Z-slice) + if(hasPrevSlice && prevSlice[nativeInSlice] == 0) + { + return true; + } + // +remappedY neighbour (next Z-slice) + if(hasNextSlice && nextSlice[nativeInSlice] == 0) + { + return true; + } + } + } + + return false; +} + +/** + * @brief Checks whether a voxel at (x,y) within a Z-slice is on the surface + * in 3D. Uses the three-slice rolling window: prevSlice, curSlice, nextSlice. + */ +bool IsPointASurfaceFeature3D(usize x, usize y, usize z, usize xPoints, usize yPoints, usize zPoints, bool markFeature0Neighbors, const std::vector& prevSlice, + const std::vector& curSlice, const std::vector& nextSlice) +{ + if(x <= 0 || x >= xPoints - 1) + { + return true; + } + if(y <= 0 || y >= yPoints - 1) + { + return true; + } + if(z <= 0 || z >= zPoints - 1) + { + return true; + } + + if(markFeature0Neighbors) + { + const usize inSlice = y * xPoints + x; + + // -X + if(curSlice[inSlice - 1] == 0) + { + return true; + } + // +X + if(curSlice[inSlice + 1] == 0) + { + return true; + } + // -Y + if(curSlice[inSlice - xPoints] == 0) + { + return true; + } + // +Y + if(curSlice[inSlice + xPoints] == 0) + { + return true; + } + // -Z (previous slice) + if(prevSlice[inSlice] == 0) + { + return true; + } + // +Z (next slice) + if(nextSlice[inSlice] == 0) + { + return true; + } + } + + return false; +} +} // namespace + +// ----------------------------------------------------------------------------- +ComputeSurfaceFeaturesScanline::ComputeSurfaceFeaturesScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const ComputeSurfaceFeaturesInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +ComputeSurfaceFeaturesScanline::~ComputeSurfaceFeaturesScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief Identifies surface features using Z-slice bulk I/O with a rolling window. + * OOC path: reads featureIds one Z-slice at a time via copyIntoBuffer(), + * keeping 3 slices resident (prev/cur/next) for cross-slice neighbour access. + * Handles both 3D and 2D geometries with coordinate remapping. + */ +Result<> ComputeSurfaceFeaturesScanline::operator()() +{ + const auto pMarkFeature0NeighborsValue = m_InputValues->MarkFeature0Neighbors; + const auto pFeatureGeometryPathValue = m_InputValues->InputImageGeometryPath; + const auto pFeatureIdsArrayPathValue = m_InputValues->FeatureIdsPath; + const auto pFeaturesAttributeMatrixPathValue = m_InputValues->FeatureAttributeMatrixPath; + const auto pSurfaceFeaturesArrayPathValue = pFeaturesAttributeMatrixPathValue.createChildPath(m_InputValues->SurfaceFeaturesArrayName); + + const auto& featureIdsArray = m_DataStructure.getDataRefAs(pFeatureIdsArrayPathValue); + + auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, pFeaturesAttributeMatrixPathValue, featureIdsArray, false, m_MessageHandler); + if(validateNumFeatResult.invalid()) + { + return validateNumFeatResult; + } + + const auto& featureGeometry = m_DataStructure.getDataRefAs(pFeatureGeometryPathValue); + auto& featureIds = m_DataStructure.getDataAs(pFeatureIdsArrayPathValue)->getDataStoreRef(); + auto& surfaceFeatures = m_DataStructure.getDataAs(pSurfaceFeaturesArrayPathValue)->getDataStoreRef(); + + // Cache the small feature-level surfaceFeatures array locally to avoid + // per-voxel OOC operator[] in the hot Z-Y-X loop. + const usize numFeatures = surfaceFeatures.getNumberOfTuples(); + std::vector localSurfaceFeatures(numFeatures, 0); + surfaceFeatures.copyIntoBuffer(0, nonstd::span(localSurfaceFeatures.data(), numFeatures)); + + const usize xPoints = featureGeometry.getNumXCells(); + const usize yPoints = featureGeometry.getNumYCells(); + const usize zPoints = featureGeometry.getNumZCells(); + const usize geometryDimensionality = featureGeometry.getDimensionality(); + + // For 2D geometries, compute the remapped dimensions + usize remappedXPoints = 0; + usize remappedYPoints = 0; + if(geometryDimensionality == 2) + { + if(xPoints == 1) + { + remappedXPoints = yPoints; + remappedYPoints = zPoints; + } + else if(yPoints == 1) + { + remappedXPoints = xPoints; + remappedYPoints = zPoints; + } + else // zPoints == 1 + { + remappedXPoints = xPoints; + remappedYPoints = yPoints; + } + } + + // Z-slice rolling window for bulk I/O + const usize sliceSize = yPoints * xPoints; + std::vector prevSlice(sliceSize, 0); + std::vector curSlice(sliceSize, 0); + std::vector nextSlice(sliceSize, 0); + + // Load initial slices + featureIds.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + if(zPoints > 1) + { + featureIds.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + + for(usize z = 0; z < zPoints; z++) + { + if(m_ShouldCancel) + { + return {}; + } + for(usize y = 0; y < yPoints; y++) + { + for(usize x = 0; x < xPoints; x++) + { + const usize inSlice = y * xPoints + x; + const int32 gNum = curSlice[inSlice]; + if(gNum != 0 && !localSurfaceFeatures[gNum]) + { + if(geometryDimensionality == 3) + { + if(IsPointASurfaceFeature3D(x, y, z, xPoints, yPoints, zPoints, pMarkFeature0NeighborsValue, prevSlice, curSlice, nextSlice)) + { + localSurfaceFeatures[gNum] = 1; + } + } + else if(geometryDimensionality == 2) + { + // Remap native 3D coordinates to 2D based on degenerate dimension + usize remappedX = 0; + usize remappedY = 0; + bool degenerateZ = false; + if(xPoints == 1) + { + remappedX = y; + remappedY = z; + } + else if(yPoints == 1) + { + remappedX = x; + remappedY = z; + } + else // zPoints == 1 + { + remappedX = x; + remappedY = y; + degenerateZ = true; + } + + if(IsPointASurfaceFeature2D(remappedX, remappedY, remappedXPoints, remappedYPoints, pMarkFeature0NeighborsValue, curSlice, prevSlice, nextSlice, inSlice, z > 0, z + 1 < zPoints, + degenerateZ)) + { + localSurfaceFeatures[gNum] = 1; + } + } + else + { + return MakeErrorResult(-1000, fmt::format("Image Geometry at path '{}' must be either 3D or 2D", pFeatureGeometryPathValue.toString())); + } + } + } + } + + // Shift the rolling window + std::swap(prevSlice, curSlice); + std::swap(curSlice, nextSlice); + if(z + 2 < zPoints) + { + featureIds.copyIntoBuffer((z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + } + + // Write cached results back to the OOC store + surfaceFeatures.copyFromBuffer(0, nonstd::span(localSurfaceFeatures.data(), numFeatures)); + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp new file mode 100644 index 0000000000..3e757e7c01 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp @@ -0,0 +1,39 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct ComputeSurfaceFeaturesInputValues; + +/** + * @class ComputeSurfaceFeaturesScanline + * @brief Out-of-core algorithm for ComputeSurfaceFeatures. Uses chunk-sequential 3D iteration + * with 2D coordinate remapping for degenerate dimensions, ensuring sequential disk I/O + * on ZarrStore-backed arrays. Selected by DispatchAlgorithm when any input array is + * backed by ZarrStore. + */ +class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesScanline +{ +public: + ComputeSurfaceFeaturesScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceFeaturesInputValues* inputValues); + ~ComputeSurfaceFeaturesScanline() noexcept; + + ComputeSurfaceFeaturesScanline(const ComputeSurfaceFeaturesScanline&) = delete; + ComputeSurfaceFeaturesScanline(ComputeSurfaceFeaturesScanline&&) noexcept = delete; + ComputeSurfaceFeaturesScanline& operator=(const ComputeSurfaceFeaturesScanline&) = delete; + ComputeSurfaceFeaturesScanline& operator=(ComputeSurfaceFeaturesScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp index 6825b65cda..38ec3da490 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp @@ -46,12 +46,17 @@ class CropImageGeomDataArray protected: void convert() const { - size_t numComps = m_OldCellStore.getNumberOfComponents(); + usize numComps = m_OldCellStore.getNumberOfComponents(); m_NewCellStore.fill(static_cast(-1)); auto srcDims = m_SrcImageGeom.getDimensions(); + // Copy one X-row at a time using bulk I/O + const uint64 rowTuples = m_Bounds[1] - m_Bounds[0]; + const usize rowElements = rowTuples * numComps; + auto rowBuffer = std::make_unique(rowElements); + uint64 destTupleIndex = 0; for(uint64 zIndex = m_Bounds[4]; zIndex < m_Bounds[5]; zIndex++) { @@ -61,15 +66,10 @@ class CropImageGeomDataArray } for(uint64 yIndex = m_Bounds[2]; yIndex < m_Bounds[3]; yIndex++) { - for(uint64 xIndex = m_Bounds[0]; xIndex < m_Bounds[1]; xIndex++) - { - uint64 srcIndex = (srcDims[0] * srcDims[1] * zIndex) + (srcDims[0] * yIndex) + xIndex; - for(size_t compIndex = 0; compIndex < numComps; compIndex++) - { - m_NewCellStore.setValue(destTupleIndex * numComps + compIndex, m_OldCellStore.getValue(srcIndex * numComps + compIndex)); - } - destTupleIndex++; - } + uint64 srcRowStart = (srcDims[0] * srcDims[1] * zIndex) + (srcDims[0] * yIndex) + m_Bounds[0]; + m_OldCellStore.copyIntoBuffer(srcRowStart * numComps, nonstd::span(rowBuffer.get(), rowElements)); + m_NewCellStore.copyFromBuffer(destTupleIndex * numComps, nonstd::span(rowBuffer.get(), rowElements)); + destTupleIndex += rowTuples; } } } @@ -223,7 +223,7 @@ Result<> CropImageGeometry::operator()() // so that the updating of the Feature level data can happen. We do a bit of // under-the-covers where we actually remove the existing array that preflight // created, so we can use the convenience of the DataArray.deepCopy() function. - for(size_t index = 0; index < sourceFeatureDataPaths.size(); index++) + for(usize index = 0; index < sourceFeatureDataPaths.size(); index++) { DataObject* dataObject = m_DataStructure.getData(sourceFeatureDataPaths[index]); if(dataObject->getDataObjectType() == DataObject::Type::DataArray) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp new file mode 100644 index 0000000000..81ff9eac6a --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp @@ -0,0 +1,29 @@ +#include "DBSCAN.hpp" + +#include "DBSCANDirect.hpp" +#include "DBSCANScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +DBSCAN::DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +DBSCAN::~DBSCAN() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> DBSCAN::operator()() +{ + auto* clusteringArray = m_DataStructure.getDataAs(m_InputValues->ClusteringArrayPath); + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + return DispatchAlgorithm({clusteringArray, featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp new file mode 100644 index 0000000000..f2e6d9e070 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp @@ -0,0 +1,58 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ChoicesParameter.hpp" +#include "simplnx/Utilities/ClusteringUtilities.hpp" + +#include + +namespace nx::core +{ +struct SIMPLNXCORE_EXPORT DBSCANInputValues +{ + DataPath ClusteringArrayPath; + DataPath MaskArrayPath; + DataPath FeatureIdsArrayPath; + float32 Epsilon; + int32 MinPoints; + ClusterUtilities::DistanceMetric DistanceMetric; + DataPath FeatureAM; + ChoicesParameter::ValueType ParseOrder; + std::mt19937_64::result_type Seed; +}; + +/** + * @class + */ +class SIMPLNXCORE_EXPORT DBSCAN +{ +public: + DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues); + ~DBSCAN() noexcept; + + DBSCAN(const DBSCAN&) = delete; + DBSCAN(DBSCAN&&) noexcept = delete; + DBSCAN& operator=(const DBSCAN&) = delete; + DBSCAN& operator=(DBSCAN&&) noexcept = delete; + + enum ParseOrder + { + LowDensityFirst, + Random, + SeededRandom + }; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const DBSCANInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp index 35547bcbe5..69208d4d33 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp @@ -1,5 +1,7 @@ #include "DBSCANDirect.hpp" +#include "DBSCAN.hpp" + #include "simplnx/Common/Range.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -16,78 +18,37 @@ namespace { /** * Implementation derived from: https://yliu.site/pub/GDCF_PR2019.pdf - * Citation: - * Thapana Boonchoo, Xiang Ao, Yang Liu, Weizhong Zhao, Fuzhen Zhuang, Qing He, - * Grid-based DBSCANDirect: Indexing and inference, - * https://doi.org/10.1016/j.patcog.2019.01.034. * - * Definitions: - * - Core Grid - A grid that contains more than the minPoints - * - Border Grid - A grid that contains less than the minPoints, - * but is density-reachable from an existing cluster - * - Noise Grid - A grid with less than minPoints, and is unreachable - * from a valid cluster + * In-core variant: uses direct per-element getValue()/operator[] access. */ -/** - * @brief This object packs a sparse matrix into a vector of uint8s. It - * represents a singular dimension and must be used in tandem with another - * from each dimension in the input array. - * - * It stores a form of adjacency matrix that is utilized as a look up - * table for Nearest Neighbor queries. - */ struct GridBitMap { std::vector gridTable = {}; usize numPositions = 0; - - // This value represents the number of bytes allocated - // to each row in the map - // Reason: stored to speed up indexing and access usize rowLength = 0; }; -/** - * @brief This object contains a function for creating GridBitMaps that handles - * all the setup for the object. The decision to make it a Factory object comes - * from the need to assemble multiple depending on the dimensions of input. - */ struct GridBitMapFactory { - /** - * Note here we can pack it slightly tighter by not adding buffers at the end of each row (for grid counts not divisible by 8) - * but this will make calculations more difficult and costly during neighbor search - * At most this saves 7/8s of a byte per dimension worth of space for significant calculation - * and parse cost incursion - */ static GridBitMap createGridBitMap(usize numGrids, usize numPositons) { GridBitMap gridBitMap = {}; usize bitPackSize = numGrids / 8; - bitPackSize += static_cast((numGrids % 8 > 0)); // Cast to avoid if/else branch + bitPackSize += static_cast((numGrids % 8 > 0)); gridBitMap.numPositions = numPositons; gridBitMap.rowLength = bitPackSize; - gridBitMap.gridTable.resize(bitPackSize * numPositons); return gridBitMap; } }; -/** - * @brief HyperGridBitMap is the superclass for two specializations of 2D and 3D. These - * read an input array to define a relevant regular grid. It bins the values in the input - * array into cells in the grid then compresses the stored grids to just the ones containing - * points (gridVoxels). It then builds several psuedo-adjacency maps to preserve the spatial - * relationship between grids along each dimension. - */ class HyperGridBitMap { public: - // Grid Cells std::vector> gridVoxels = {}; protected: @@ -110,29 +71,28 @@ class HyperGridBitMap3D : public HyperGridBitMap const std::unique_ptr& mask) : HyperGridBitMap() { - ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); + const usize numTuples = inputArray.getNumberOfTuples(); + const usize numComps = inputArray.getNumberOfComponents(); messageHelper.sendMessage(" - Determining bounds..."); - // Load array bounds + // Load array bounds using direct per-element access std::array bounds = {std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN()}; - for(usize i = 0; i < inputArray.getNumberOfTuples(); i++) + for(usize tup = 0; tup < numTuples; tup++) { if(shouldCancel) { return; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Finding Bounds || {:.2f}% Complete", CalculatePercentComplete(i, inputArray.getNumberOfTuples())); }); - - if(!mask->isTrue(i)) + if(!mask->isTrue(tup)) { continue; } - auto xVal = static_cast(inputArray.getValue((i * 3) + 0)); - auto yVal = static_cast(inputArray.getValue((i * 3) + 1)); - auto zVal = static_cast(inputArray.getValue((i * 3) + 2)); + auto xVal = static_cast(inputArray[tup * numComps + 0]); + auto yVal = static_cast(inputArray[tup * numComps + 1]); + auto zVal = static_cast(inputArray[tup * numComps + 2]); bounds[0] = std::isnan(bounds[0]) ? xVal : std::min(bounds[0], xVal); bounds[1] = std::isnan(bounds[1]) ? yVal : std::min(bounds[1], yVal); @@ -164,30 +124,25 @@ class HyperGridBitMap3D : public HyperGridBitMap std::vector> positions = {}; // Build a set of non-empty grids and temporarily store their positions { - usize numTup = inputArray.getNumberOfTuples(); std::vector grids(std::accumulate(dims.cbegin(), dims.cend(), static_cast(1), std::multiplies<>()), false); - // Find num grid cells - for(usize tup = 0; tup < numTup; tup++) + // Find num grid cells - direct access pass + for(usize tup = 0; tup < numTuples; tup++) { if(shouldCancel) { return; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Binning || {:.2f}% Complete", CalculatePercentComplete(tup, numTup * 2)); }); - if(!mask->isTrue(tup)) { continue; } - // Determine the voxel - usize pointIdx = tup * inputArray.getNumberOfComponents(); - usize xPos = std::floor((inputArray.getValue(pointIdx + 0) - origin[0]) / spacing[0]); - usize yPos = std::floor((inputArray.getValue(pointIdx + 1) - origin[1]) / spacing[1]); - usize zPos = std::floor((inputArray.getValue(pointIdx + 2) - origin[2]) / spacing[2]); - usize bin = (zPos * dims[1] * dims[0]) + (yPos * dims[0]) + xPos; + usize xPos = std::floor((static_cast(inputArray[tup * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(inputArray[tup * numComps + 1]) - origin[1]) / spacing[1]); + usize zPos = std::floor((static_cast(inputArray[tup * numComps + 2]) - origin[2]) / spacing[2]); + usize bin = (zPos * dims[1] * dims[0]) + (yPos * dims[0]) + xPos; grids[bin] = true; } @@ -203,54 +158,45 @@ class HyperGridBitMap3D : public HyperGridBitMap gridMap[i] = activeGridCount; activeGridCount++; - std::array position = {}; // Trivially copyable + std::array position = {}; position[2] = i / zSize; - usize zRemdr = i % zSize; // Modern compilers will extract the result from previous instruction + usize zRemdr = i % zSize; position[1] = zRemdr / ySize; - position[0] = zRemdr % ySize; // Modern compilers will extract the result from previous instruction + position[0] = zRemdr % ySize; positions.push_back(position); } } gridVoxels = std::vector>(activeGridCount, std::vector(0)); - // Fill grid cells - for(usize tup = 0; tup < numTup; tup++) + // Fill grid cells - direct access pass + for(usize tup = 0; tup < numTuples; tup++) { if(shouldCancel) { return; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Binning || {:.2f}% Complete", CalculatePercentComplete(numTup + tup, numTup * 2)); }); - if(!mask->isTrue(tup)) { continue; } - // Determine the voxel - usize pointIdx = tup * inputArray.getNumberOfComponents(); - usize xPos = std::floor((inputArray.getValue(pointIdx + 0) - origin[0]) / spacing[0]); - usize yPos = std::floor((inputArray.getValue(pointIdx + 1) - origin[1]) / spacing[1]); - usize zPos = std::floor((inputArray.getValue(pointIdx + 2) - origin[2]) / spacing[2]); - usize bin = (zPos * dims[1] * dims[0]) + (yPos * dims[0]) + xPos; + usize xPos = std::floor((static_cast(inputArray[tup * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(inputArray[tup * numComps + 1]) - origin[1]) / spacing[1]); + usize zPos = std::floor((static_cast(inputArray[tup * numComps + 2]) - origin[2]) / spacing[2]); + usize bin = (zPos * dims[1] * dims[0]) + (yPos * dims[0]) + xPos; gridVoxels[gridMap[bin]].push_back(tup); } } // End of filling non-empty grids and positions vector - // Pack down memory further (run outside block to clear mem faster) + // Pack down memory further for(auto& grid : gridVoxels) { grid.shrink_to_fit(); } messageHelper.sendMessage(" - Generating adjacency matrix for search..."); - /** - * This could be modified to 3 passes on the positions vector with custom predicates and ths std::sort function, - * but we are sacrificing space for speed, because its a subset of a known predefined grid - */ - // Make sets to bin grids std::set xSet = {}; std::set ySet = {}; std::set zSet = {}; @@ -267,7 +213,6 @@ class HyperGridBitMap3D : public HyperGridBitMap return; } - // Set up hyper bit map xTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), xSet.size()); yTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), ySet.size()); zTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), zSet.size()); @@ -277,8 +222,6 @@ class HyperGridBitMap3D : public HyperGridBitMap return; } - // Not the most efficient fill but due to the random access nature of position - // we can't load one consecutive mask for(usize gridId = 0; gridId < positions.size(); gridId++) { usize relativeGridBytePos = gridId / 8; @@ -322,28 +265,26 @@ class HyperGridBitMap2D : public HyperGridBitMap const std::unique_ptr& mask) : HyperGridBitMap() { - ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); + const usize numTuples = inputArray.getNumberOfTuples(); + const usize numComps = inputArray.getNumberOfComponents(); - // Load array bounds + // Load array bounds using direct per-element access std::array bounds = {std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN()}; - for(usize i = 0; i < inputArray.getNumberOfTuples(); i++) + for(usize tup = 0; tup < numTuples; tup++) { if(shouldCancel) { return; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Finding Bounds || {:.2f}% Complete", CalculatePercentComplete(i, inputArray.getNumberOfTuples())); }); - - if(!mask->isTrue(i)) + if(!mask->isTrue(tup)) { continue; } - // Determine the voxel - auto xVal = static_cast(inputArray.getValue((i * 2) + 0)); - auto yVal = static_cast(inputArray.getValue((i * 2) + 1)); + auto xVal = static_cast(inputArray[tup * numComps + 0]); + auto yVal = static_cast(inputArray[tup * numComps + 1]); bounds[0] = std::isnan(bounds[0]) ? xVal : std::min(bounds[0], xVal); bounds[1] = std::isnan(bounds[1]) ? yVal : std::min(bounds[1], yVal); @@ -371,30 +312,24 @@ class HyperGridBitMap2D : public HyperGridBitMap std::vector> positions = {}; // Build a set of non-empty grids and temporarily store their positions { - usize numTup = inputArray.getNumberOfTuples(); std::vector grids(std::accumulate(dims.cbegin(), dims.cend(), static_cast(1), std::multiplies<>()), false); - // Find num grid cells - for(usize tup = 0; tup < numTup; tup++) + // Find num grid cells - direct access pass + for(usize tup = 0; tup < numTuples; tup++) { if(shouldCancel) { return; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Binning || {:.2f}% Complete", CalculatePercentComplete(tup, numTup * 2)); }); - if(!mask->isTrue(tup)) { continue; } - // Determine the voxel - usize pointIdx = tup * inputArray.getNumberOfComponents(); - usize xPos = std::floor((inputArray.getValue(pointIdx + 0) - origin[0]) / spacing[0]); - usize yPos = std::floor((inputArray.getValue(pointIdx + 1) - origin[1]) / spacing[1]); + usize xPos = std::floor((static_cast(inputArray[tup * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(inputArray[tup * numComps + 1]) - origin[1]) / spacing[1]); usize bin = (yPos * dims[0]) + xPos; - grids[bin] = true; } @@ -410,51 +345,42 @@ class HyperGridBitMap2D : public HyperGridBitMap gridMap[i] = activeGridCount; activeGridCount++; - std::array position = {}; // Trivially copyable + std::array position = {}; position[1] = i / ySize; - position[0] = i % ySize; // Modern compilers will extract the result from previous instruction + position[0] = i % ySize; positions.push_back(position); } } gridVoxels = std::vector>(activeGridCount, std::vector(0)); - // Fill grid cells - for(usize tup = 0; tup < numTup; tup++) + // Fill grid cells - direct access pass + for(usize tup = 0; tup < numTuples; tup++) { if(shouldCancel) { return; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Binning || {:.2f}% Complete", CalculatePercentComplete(numTup + tup, numTup * 2)); }); - if(!mask->isTrue(tup)) { continue; } - // Determine the voxel - usize pointIdx = tup * inputArray.getNumberOfComponents(); - usize xPos = std::floor((inputArray.getValue(pointIdx + 0) - origin[0]) / spacing[0]); - usize yPos = std::floor((inputArray.getValue(pointIdx + 1) - origin[1]) / spacing[1]); - usize bin = (yPos * dims[0]) + xPos; + usize xPos = std::floor((static_cast(inputArray[tup * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(inputArray[tup * numComps + 1]) - origin[1]) / spacing[1]); + usize bin = (yPos * dims[0]) + xPos; gridVoxels[gridMap[bin]].push_back(tup); } } // End of filling non-empty grids and positions vector - // Pack down memory further (run outside block to clear mem faster) + // Pack down memory further for(auto& grid : gridVoxels) { grid.shrink_to_fit(); } messageHelper.sendMessage(" - Generating adjacency matrix for search..."); - /** - * This could be modified to 2 passes on the positions vector with custom predicates and ths std::sort function, - * but we are sacrificing space for speed, because its a subset of a known predefined grid - */ - // Make sets to bin grids std::set xSet = {}; std::set ySet = {}; @@ -469,7 +395,6 @@ class HyperGridBitMap2D : public HyperGridBitMap return; } - // Set up hyper bit map xTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), xSet.size()); yTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), ySet.size()); @@ -478,8 +403,6 @@ class HyperGridBitMap2D : public HyperGridBitMap return; } - // Not the most efficient fill but due to the random access nature of position - // we can't load one consecutive mask for(usize gridId = 0; gridId < positions.size(); gridId++) { usize relativeGridBytePos = gridId / 8; @@ -506,11 +429,9 @@ void SearchTablePositions(std::vector& outputGridMask, usize searchSpace, { std::vector tempGridMask(selectedTable.rowLength, 0); - // Find indices to search space usize xStart = (targetPosition < searchSpace) ? 0 : targetPosition - searchSpace; usize xEnd = (targetPosition + searchSpace < selectedTable.numPositions) ? targetPosition + searchSpace + 1 : selectedTable.numPositions; - // Store all grids in the positions within the dimensional search space for(usize pos = xStart; pos < xEnd; pos++) { for(usize i = 0; i < selectedTable.rowLength; i++) @@ -519,8 +440,6 @@ void SearchTablePositions(std::vector& outputGridMask, usize searchSpace, } } - // Narrow down search by overlaying this dimension's search space - // onto previous dimensions search space for(usize i = 0; i < selectedTable.rowLength; i++) { outputGridMask[i] = tempGridMask[i] & outputGridMask[i]; @@ -537,11 +456,8 @@ std::vector NeighborGridQuery(usize targetGridId, const HGBPT& hyperGridB std::vector neighborGridIds = {}; - // check adjacent positions in the table by sqrt(Dimensions) for grid ids std::vector finalGridMask(hyperGridBitMap.xTable.rowLength, std::numeric_limits::max()); - // The search loops to find xyzPos can be cut if we opt to store the - // positions for each grid cell within each cell or in a separate vector usize relativeGridBytePos = targetGridId / 8; uint8 bitGridOffset = targetGridId % 8; @@ -612,18 +528,14 @@ std::vector NeighborGridQuery(usize targetGridId, const HGBPT& hyperGridB struct ClusterNode { - int32 clusterId; - usize parent; + int32 clusterId = 0; + usize parent = 0; }; struct ClusterForest { std::vector clusterForestNodes = {}; - /** - * @brief Primes the cluster forest object - * @param numGrids the total number of gridVoxels containing points (not just core grids) - */ void initialize(usize numGrids) { clusterForestNodes.resize(numGrids); @@ -645,30 +557,11 @@ struct ClusterForest return findClusterRoot(clusterForestNodes[gridId].parent); } - /** - * @brief Checks if grids are already in the same cluster - * Note: NO BOUNDS CHECKING - * @param pGridId a valid grid id - * @param qGridId a valid grid id - * @return bool if true they are in the same cluster - */ bool infer(usize pGridId, usize qGridId) { return findClusterRoot(pGridId) == findClusterRoot(qGridId); } - /** - * @brief This function merges every supplied grid into the cluster with the - * lowest cluster id. - * - * Note: DO NOT PASS IN A BORDER GRID THAT HAS ITSELF AS THE PARENT. This will - * collapse all your clusters into unlabeled category. Ids to border grids that - * have a valid Core Grid parent are fine. - * - * The best way to avoid collapse is never make a border grid with itself as - * the parent, the parent of another border grid - * @param gridIds - a vector of ids representing grids with valid parents to be merged - */ void mergeLRC(const std::vector& gridIds) { if(gridIds.size() < 2) @@ -700,6 +593,9 @@ struct ClusterForest } }; +/** + * @brief In-core GDCF: uses direct operator[] access for canMerge. + */ template class GDCF { @@ -716,10 +612,9 @@ class GDCF { } - Result<> cluster(usize minPoints, DBSCANDirect::ParseOrder parseOrder, std::mt19937_64::result_type seed = std::mt19937_64::default_seed) + Result<> cluster(usize minPoints, DBSCAN::ParseOrder parseOrder, std::mt19937_64::result_type seed = std::mt19937_64::default_seed) { m_MessageHelper.sendMessage(" - Identifying core grids..."); - // Identify Core Grids std::vector coreGridIds = {}; for(usize i = 0; i < hyperGridBitMap.gridVoxels.size(); i++) { @@ -738,41 +633,36 @@ class GDCF return {}; } - // Sort Grids to reduce bias m_MessageHelper.sendMessage(" - Sorting grids according to supplied parse order..."); switch(parseOrder) { - case DBSCANDirect::ParseOrder::LowDensityFirst: { + case DBSCAN::ParseOrder::LowDensityFirst: { QuickSortGrids(coreGridIds, 0, coreGridIds.size() - 1); break; } - case DBSCANDirect::ParseOrder::Random: { + case DBSCAN::ParseOrder::Random: { std::mt19937_64 gen(seed); std::uniform_real_distribution dist(0, 1); auto maxIdx = static_cast(coreGridIds.size() - 1); - //--- Shuffle elements by randomly exchanging each with one other. for(usize i = 1; i < coreGridIds.size(); i++) { - auto r = static_cast(std::floor(dist(gen) * maxIdx)); // Random remaining position. - + auto r = static_cast(std::floor(dist(gen) * maxIdx)); std::swap(coreGridIds[i], coreGridIds[r]); } break; } - case DBSCANDirect::SeededRandom: { + case DBSCAN::SeededRandom: { std::mt19937_64 gen(seed); std::uniform_real_distribution dist(0, 1); auto maxIdx = static_cast(coreGridIds.size() - 1); - //--- Shuffle elements by randomly exchanging each with one other. for(usize i = 1; i < coreGridIds.size(); i++) { - auto r = static_cast(std::floor(dist(gen) * maxIdx)); // Random remaining position. - + auto r = static_cast(std::floor(dist(gen) * maxIdx)); std::swap(coreGridIds[i], coreGridIds[r]); } @@ -786,7 +676,6 @@ class GDCF } m_MessageHelper.sendMessage("Identifying Qualifying Independent Clusters:"); - ThrottledMessenger throttledMessenger = m_MessageHelper.createThrottledMessenger(); clusterForest.initialize(hyperGridBitMap.gridVoxels.size()); for(usize i = 0; i < coreGridIds.size(); i++) { @@ -795,37 +684,25 @@ class GDCF return {}; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Identifying clusters || {:.2f}% Complete", CalculatePercentComplete(i, coreGridIds.size())); }); - std::vector neighborGrids = NeighborGridQuery(coreGridIds[i], hyperGridBitMap); std::vector cluster = {}; cluster.push_back(coreGridIds[i]); for(const usize gridId : neighborGrids) { - // If true they are in the same cluster if(clusterForest.infer(coreGridIds[i], gridId)) { continue; } - // Check if a point in neighbor grid is density reachable if(canMerge(coreGridIds[i], gridId)) { - // Check if it's a border grid and check if its unvisited if(hyperGridBitMap.gridVoxels[gridId].size() < minPoints && clusterForest.clusterForestNodes[gridId].parent == gridId) { - // Border grids can not be their own cluster, which means this - // is unvisited currently so merge it into the current cluster clusterForest.clusterForestNodes[gridId].parent = coreGridIds[i]; } else { - // Either this is a density-reachable core grid - // OR - // This border grid belongs to another cluster, but the fact it is - // reachable here means that the two clusters are one and need to - // be merged cluster.push_back(gridId); } } @@ -844,7 +721,6 @@ class GDCF operations = 0; for(usize i = 0; i < hyperGridBitMap.gridVoxels.size(); i++) { - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Expanding clusters || {:.2f}% Complete", CalculatePercentComplete(i, hyperGridBitMap.gridVoxels.size())); }); if(m_ShouldCancel) { return {}; @@ -865,13 +741,10 @@ class GDCF { usize activeParent = clusterForest.findClusterRoot(i); usize neighborGridParent = clusterForest.findClusterRoot(gridId); - // Check if search grid has been visited if(activeParent == i) { if(hyperGridBitMap.gridVoxels[gridId].size() < minPoints && neighborGridParent == gridId) { - // Border grids can not be their own cluster, which means this - // is unvisited currently; continue; } clusterForest.clusterForestNodes[i].parent = neighborGridParent; @@ -890,7 +763,6 @@ class GDCF } else { - // Infer returning false means that they can't have the same cluster id so must be greater than clusterForest.clusterForestNodes[activeParent].parent = neighborGridParent; } } @@ -903,7 +775,6 @@ class GDCF } while(operations > 0); m_MessageHelper.sendMessage(" - Cleaning up cluster identifiers..."); - // clean up cluster forest std::vector clusters = {}; for(usize i = 0; i < clusterForest.clusterForestNodes.size(); i++) { @@ -911,12 +782,10 @@ class GDCF { if(hyperGridBitMap.gridVoxels[i].size() >= minPoints) { - // Only core nodes can be their own parent clusters.push_back(i); } else { - // grid unreachable, label noise clusterForest.clusterForestNodes[i].clusterId = 0; } } @@ -937,8 +806,6 @@ class GDCF return MakeWarningVoidResult(-85640, "No clusters detected - Consider reducing number of required points (`Minimum Points`) or increasing acceptable distance (`Epsilon`)."); } - ThrottledMessenger throttledMessenger = m_MessageHelper.createThrottledMessenger(); - // label fIdsDataStore.fill(0); for(usize gridIdx = 0; gridIdx < hyperGridBitMap.gridVoxels.size(); gridIdx++) { @@ -947,8 +814,6 @@ class GDCF return {}; } - throttledMessenger.sendThrottledMessage([&]() { return fmt::format(" - Labeling || {:.2f}% Complete", CalculatePercentComplete(gridIdx, hyperGridBitMap.gridVoxels.size())); }); - int32 featureId = clusterForest.clusterForestNodes[clusterForest.findClusterRoot(gridIdx)].clusterId; for(usize pointIdx : hyperGridBitMap.gridVoxels[gridIdx]) { @@ -964,7 +829,7 @@ class GDCF ClusterForest clusterForest = {}; - float32 m_Epsilon; + float32 m_Epsilon = 0.0f; const AbstractDataStore& m_InputDataStore; ClusterUtilities::DistanceMetric m_DistMetric; const std::atomic_bool& m_ShouldCancel; @@ -1010,11 +875,11 @@ class GDCF usize next = ProcessSection(sorted, begin, end); - // Recurse QuickSortGrids(sorted, begin, next); QuickSortGrids(sorted, next + 1, end); } + // In-core path: direct random access via operator[] is fast bool canMerge(usize pGridId, usize qGridId) { for(usize pPointId : hyperGridBitMap.gridVoxels[pGridId]) @@ -1028,7 +893,6 @@ class GDCF } } } - return false; } }; @@ -1046,11 +910,9 @@ Result<> RunAlgorithm(const DBSCANInputValues* inputValues, const AbstractDataSt } messageHelper.sendMessage("Clustering:"); - Result<> result = algorithm.cluster(inputValues->MinPoints, static_cast(inputValues->ParseOrder), inputValues->Seed); + Result<> result = algorithm.cluster(inputValues->MinPoints, static_cast(inputValues->ParseOrder), inputValues->Seed); if(result.invalid() || !result.warnings().empty()) { - // If the result has warnings in it the cluster forest is - // ill-formed, so skip labeling step. return result; } @@ -1089,7 +951,7 @@ struct DBSCANDirectFunctor } // namespace // ----------------------------------------------------------------------------- -DBSCANDirect::DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues) +DBSCANDirect::DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const DBSCANInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -1101,6 +963,9 @@ DBSCANDirect::DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageH DBSCANDirect::~DBSCANDirect() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief In-core DBSCAN execution using direct per-element access. + */ Result<> DBSCANDirect::operator()() { MessageHelper messageHelper(m_MessageHandler); @@ -1114,8 +979,6 @@ Result<> DBSCANDirect::operator()() maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); } catch(const std::out_of_range& exception) { - // This really should NOT be happening as the path was verified during preflight BUT we may be calling this from - // somewhere else that is NOT going through the normal nx::core::IFilter API of Preflight and Execute std::string message = fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString()); return MakeErrorResult(-54060, message); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp index 16f836ac4c..2f8edee2f0 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp @@ -2,36 +2,23 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ChoicesParameter.hpp" -#include "simplnx/Utilities/ClusteringUtilities.hpp" - -#include namespace nx::core { -struct SIMPLNXCORE_EXPORT DBSCANInputValues -{ - DataPath ClusteringArrayPath; - DataPath MaskArrayPath; - DataPath FeatureIdsArrayPath; - float32 Epsilon; - int32 MinPoints; - ClusterUtilities::DistanceMetric DistanceMetric; - DataPath FeatureAM; - ChoicesParameter::ValueType ParseOrder; - std::mt19937_64::result_type Seed; -}; +struct DBSCANInputValues; /** - * @class + * @class DBSCANDirect + * @brief In-core algorithm for DBSCAN. Uses direct per-element getValue()/operator[] + * access for grid construction and canMerge distance computation. Selected by + * DispatchAlgorithm when all input arrays are backed by in-memory DataStore. */ class SIMPLNXCORE_EXPORT DBSCANDirect { public: - DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues); + DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const DBSCANInputValues* inputValues); ~DBSCANDirect() noexcept; DBSCANDirect(const DBSCANDirect&) = delete; @@ -39,13 +26,6 @@ class SIMPLNXCORE_EXPORT DBSCANDirect DBSCANDirect& operator=(const DBSCANDirect&) = delete; DBSCANDirect& operator=(DBSCANDirect&&) noexcept = delete; - enum ParseOrder - { - LowDensityFirst, - Random, - SeededRandom - }; - Result<> operator()(); private: diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp new file mode 100644 index 0000000000..eacf822bba --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp @@ -0,0 +1,1060 @@ +#include "DBSCANScanline.hpp" + +#include "DBSCAN.hpp" + +#include "simplnx/Common/Range.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/ClusteringUtilities.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" +#include "simplnx/Utilities/MaskCompareUtilities.hpp" + +#include + +#include + +using namespace nx::core; + +namespace +{ +/** + * Implementation derived from: https://yliu.site/pub/GDCF_PR2019.pdf + * + * OOC variant: uses chunked copyIntoBuffer bulk I/O for grid construction + * and on-demand per-grid-cell reads for canMerge distance computation. + */ + +struct GridBitMap +{ + std::vector gridTable = {}; + usize numPositions = 0; + usize rowLength = 0; +}; + +struct GridBitMapFactory +{ + static GridBitMap createGridBitMap(usize numGrids, usize numPositons) + { + GridBitMap gridBitMap = {}; + + usize bitPackSize = numGrids / 8; + bitPackSize += static_cast((numGrids % 8 > 0)); + + gridBitMap.numPositions = numPositons; + gridBitMap.rowLength = bitPackSize; + gridBitMap.gridTable.resize(bitPackSize * numPositons); + + return gridBitMap; + } +}; + +class HyperGridBitMap +{ +public: + std::vector> gridVoxels = {}; + +protected: + HyperGridBitMap() = default; +}; + +class HyperGridBitMap3D : public HyperGridBitMap +{ +public: + static constexpr float32 Dimensions = 3; + + GridBitMap xTable; + GridBitMap yTable; + GridBitMap zTable; + + HyperGridBitMap3D() = delete; + + template + HyperGridBitMap3D(const std::atomic_bool& shouldCancel, const AbstractDataStore& inputArray, float32 epsilon, const std::unique_ptr& mask) + : HyperGridBitMap() + { + const usize numTuples = inputArray.getNumberOfTuples(); + const usize numComps = inputArray.getNumberOfComponents(); + constexpr usize k_ChunkTuples = 65536; + auto chunkBuf = std::make_unique(k_ChunkTuples * numComps); + + // Load array bounds using chunked bulk I/O + std::array bounds = {std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), + std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN()}; + for(usize startTup = 0; startTup < numTuples; startTup += k_ChunkTuples) + { + if(shouldCancel) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, numTuples); + const usize count = endTup - startTup; + inputArray.copyIntoBuffer(startTup * numComps, nonstd::span(chunkBuf.get(), count * numComps)); + + for(usize local = 0; local < count; local++) + { + if(!mask->isTrue(startTup + local)) + { + continue; + } + + auto xVal = static_cast(chunkBuf[local * numComps + 0]); + auto yVal = static_cast(chunkBuf[local * numComps + 1]); + auto zVal = static_cast(chunkBuf[local * numComps + 2]); + + bounds[0] = std::isnan(bounds[0]) ? xVal : std::min(bounds[0], xVal); + bounds[1] = std::isnan(bounds[1]) ? yVal : std::min(bounds[1], yVal); + bounds[2] = std::isnan(bounds[2]) ? zVal : std::min(bounds[2], zVal); + + bounds[3] = std::isnan(bounds[3]) ? xVal : std::max(bounds[3], xVal); + bounds[4] = std::isnan(bounds[4]) ? yVal : std::max(bounds[4], yVal); + bounds[5] = std::isnan(bounds[5]) ? zVal : std::max(bounds[5], zVal); + } + } + + // Grid Info - DO NOT MODIFY - basis for algorithm + float32 sideLength = epsilon / std::sqrt(Dimensions); + std::array spacing = {sideLength, sideLength, sideLength}; + + float32 buffer = sideLength; + std::array origin = {}; + origin[0] = bounds[0] - buffer; + origin[1] = bounds[1] - buffer; + origin[2] = bounds[2] - buffer; + + std::array dims = {}; + dims[0] = static_cast(((bounds[3] + buffer) - origin[0]) / spacing[0]) + 2; + dims[1] = static_cast(((bounds[4] + buffer) - origin[1]) / spacing[1]) + 2; + dims[2] = static_cast(((bounds[5] + buffer) - origin[2]) / spacing[2]) + 2; + + // Fill the BitMap + { + std::vector> positions = {}; + // Build a set of non-empty grids and temporarily store their positions + { + std::vector grids(std::accumulate(dims.cbegin(), dims.cend(), static_cast(1), std::multiplies<>()), false); + // Find num grid cells - chunked bulk I/O pass + for(usize startTup = 0; startTup < numTuples; startTup += k_ChunkTuples) + { + if(shouldCancel) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, numTuples); + const usize count = endTup - startTup; + inputArray.copyIntoBuffer(startTup * numComps, nonstd::span(chunkBuf.get(), count * numComps)); + + for(usize local = 0; local < count; local++) + { + const usize tup = startTup + local; + if(!mask->isTrue(tup)) + { + continue; + } + + usize xPos = std::floor((static_cast(chunkBuf[local * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(chunkBuf[local * numComps + 1]) - origin[1]) / spacing[1]); + usize zPos = std::floor((static_cast(chunkBuf[local * numComps + 2]) - origin[2]) / spacing[2]); + + usize bin = (zPos * dims[1] * dims[0]) + (yPos * dims[0]) + xPos; + grids[bin] = true; + } + } + usize zSize = dims[1] * dims[0]; + usize ySize = dims[0]; + usize activeGridCount = 0; + std::vector gridMap(grids.size()); + for(usize i = 0; i < grids.size(); i++) + { + if(grids[i]) + { + gridMap[i] = activeGridCount; + activeGridCount++; + + std::array position = {}; + position[2] = i / zSize; + usize zRemdr = i % zSize; + position[1] = zRemdr / ySize; + position[0] = zRemdr % ySize; + positions.push_back(position); + } + } + + gridVoxels = std::vector>(activeGridCount, std::vector(0)); + // Fill grid cells - chunked bulk I/O pass + for(usize startTup = 0; startTup < numTuples; startTup += k_ChunkTuples) + { + if(shouldCancel) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, numTuples); + const usize count = endTup - startTup; + inputArray.copyIntoBuffer(startTup * numComps, nonstd::span(chunkBuf.get(), count * numComps)); + + for(usize local = 0; local < count; local++) + { + const usize tup = startTup + local; + if(!mask->isTrue(tup)) + { + continue; + } + usize xPos = std::floor((static_cast(chunkBuf[local * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(chunkBuf[local * numComps + 1]) - origin[1]) / spacing[1]); + usize zPos = std::floor((static_cast(chunkBuf[local * numComps + 2]) - origin[2]) / spacing[2]); + + usize bin = (zPos * dims[1] * dims[0]) + (yPos * dims[0]) + xPos; + gridVoxels[gridMap[bin]].push_back(tup); + } + } + } // End of filling non-empty grids and positions vector + + // Pack down memory further + for(auto& grid : gridVoxels) + { + grid.shrink_to_fit(); + } + + std::set xSet = {}; + std::set ySet = {}; + std::set zSet = {}; + + for(const auto& position : positions) + { + xSet.insert(position[0]); + ySet.insert(position[1]); + zSet.insert(position[2]); + } + + if(shouldCancel) + { + return; + } + + xTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), xSet.size()); + yTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), ySet.size()); + zTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), zSet.size()); + + if(shouldCancel) + { + return; + } + + for(usize gridId = 0; gridId < positions.size(); gridId++) + { + usize relativeGridBytePos = gridId / 8; + uint8 bitGridOffset = gridId % 8; + + usize xPos = std::distance(xSet.begin(), xSet.find(positions[gridId][0])) * xTable.rowLength; + usize yPos = std::distance(ySet.begin(), ySet.find(positions[gridId][1])) * yTable.rowLength; + usize zPos = std::distance(zSet.begin(), zSet.find(positions[gridId][2])) * zTable.rowLength; + + usize xBytePos = xPos + relativeGridBytePos; + uint8 xMask = 1; + xMask <<= bitGridOffset; + xTable.gridTable[xBytePos] = xMask | xTable.gridTable[xBytePos]; + + usize yBytePos = yPos + relativeGridBytePos; + uint8 yMask = 1; + yMask <<= bitGridOffset; + yTable.gridTable[yBytePos] = yMask | yTable.gridTable[yBytePos]; + + usize zBytePos = zPos + relativeGridBytePos; + uint8 zMask = 1; + zMask <<= bitGridOffset; + zTable.gridTable[zBytePos] = zMask | zTable.gridTable[zBytePos]; + } + } + } +}; + +class HyperGridBitMap2D : public HyperGridBitMap +{ +public: + static constexpr float32 Dimensions = 2; + + GridBitMap xTable; + GridBitMap yTable; + + HyperGridBitMap2D() = delete; + + template + HyperGridBitMap2D(const std::atomic_bool& shouldCancel, const AbstractDataStore& inputArray, float32 epsilon, const std::unique_ptr& mask) + : HyperGridBitMap() + { + const usize numTuples = inputArray.getNumberOfTuples(); + const usize numComps = inputArray.getNumberOfComponents(); + constexpr usize k_ChunkTuples = 65536; + auto chunkBuf = std::make_unique(k_ChunkTuples * numComps); + + // Load array bounds using chunked bulk I/O + std::array bounds = {std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), std::numeric_limits::quiet_NaN(), + std::numeric_limits::quiet_NaN()}; + for(usize startTup = 0; startTup < numTuples; startTup += k_ChunkTuples) + { + if(shouldCancel) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, numTuples); + const usize count = endTup - startTup; + inputArray.copyIntoBuffer(startTup * numComps, nonstd::span(chunkBuf.get(), count * numComps)); + + for(usize local = 0; local < count; local++) + { + if(!mask->isTrue(startTup + local)) + { + continue; + } + + auto xVal = static_cast(chunkBuf[local * numComps + 0]); + auto yVal = static_cast(chunkBuf[local * numComps + 1]); + + bounds[0] = std::isnan(bounds[0]) ? xVal : std::min(bounds[0], xVal); + bounds[1] = std::isnan(bounds[1]) ? yVal : std::min(bounds[1], yVal); + + bounds[2] = std::isnan(bounds[2]) ? xVal : std::max(bounds[2], xVal); + bounds[3] = std::isnan(bounds[3]) ? yVal : std::max(bounds[3], yVal); + } + } + + // Grid Info - DO NOT MODIFY - basis for algorithm + float32 sideLength = epsilon / std::sqrt(Dimensions); + std::array spacing = {sideLength, sideLength}; + + float32 buffer = sideLength; + std::array origin = {}; + origin[0] = bounds[0] - buffer; + origin[1] = bounds[1] - buffer; + + std::array dims = {}; + dims[0] = static_cast(((bounds[2] + buffer) - origin[0]) / spacing[0]) + 2; + dims[1] = static_cast(((bounds[3] + buffer) - origin[1]) / spacing[1]) + 2; + + // Fill the BitMap + { + std::vector> positions = {}; + // Build a set of non-empty grids and temporarily store their positions + { + std::vector grids(std::accumulate(dims.cbegin(), dims.cend(), static_cast(1), std::multiplies<>()), false); + // Find num grid cells - chunked bulk I/O pass + for(usize startTup = 0; startTup < numTuples; startTup += k_ChunkTuples) + { + if(shouldCancel) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, numTuples); + const usize count = endTup - startTup; + inputArray.copyIntoBuffer(startTup * numComps, nonstd::span(chunkBuf.get(), count * numComps)); + + for(usize local = 0; local < count; local++) + { + const usize tup = startTup + local; + if(!mask->isTrue(tup)) + { + continue; + } + + usize xPos = std::floor((static_cast(chunkBuf[local * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(chunkBuf[local * numComps + 1]) - origin[1]) / spacing[1]); + + usize bin = (yPos * dims[0]) + xPos; + grids[bin] = true; + } + } + + usize ySize = dims[0]; + usize activeGridCount = 0; + std::vector gridMap(grids.size()); + for(usize i = 0; i < grids.size(); i++) + { + if(grids[i]) + { + gridMap[i] = activeGridCount; + activeGridCount++; + + std::array position = {}; + position[1] = i / ySize; + position[0] = i % ySize; + positions.push_back(position); + } + } + + gridVoxels = std::vector>(activeGridCount, std::vector(0)); + // Fill grid cells - chunked bulk I/O pass + for(usize startTup = 0; startTup < numTuples; startTup += k_ChunkTuples) + { + if(shouldCancel) + { + return; + } + const usize endTup = std::min(startTup + k_ChunkTuples, numTuples); + const usize count = endTup - startTup; + inputArray.copyIntoBuffer(startTup * numComps, nonstd::span(chunkBuf.get(), count * numComps)); + + for(usize local = 0; local < count; local++) + { + const usize tup = startTup + local; + if(!mask->isTrue(tup)) + { + continue; + } + usize xPos = std::floor((static_cast(chunkBuf[local * numComps + 0]) - origin[0]) / spacing[0]); + usize yPos = std::floor((static_cast(chunkBuf[local * numComps + 1]) - origin[1]) / spacing[1]); + + usize bin = (yPos * dims[0]) + xPos; + gridVoxels[gridMap[bin]].push_back(tup); + } + } + } // End of filling non-empty grids and positions vector + + // Pack down memory further + for(auto& grid : gridVoxels) + { + grid.shrink_to_fit(); + } + + std::set xSet = {}; + std::set ySet = {}; + + for(const auto& position : positions) + { + xSet.insert(position[0]); + ySet.insert(position[1]); + } + + if(shouldCancel) + { + return; + } + + xTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), xSet.size()); + yTable = GridBitMapFactory::createGridBitMap(gridVoxels.size(), ySet.size()); + + if(shouldCancel) + { + return; + } + + for(usize gridId = 0; gridId < positions.size(); gridId++) + { + usize relativeGridBytePos = gridId / 8; + uint8 bitGridOffset = gridId % 8; + + usize xPos = std::distance(xSet.begin(), xSet.find(positions[gridId][0])) * xTable.rowLength; + usize yPos = std::distance(ySet.begin(), ySet.find(positions[gridId][1])) * yTable.rowLength; + + usize xBytePos = xPos + relativeGridBytePos; + uint8 xMask = 1; + xMask <<= bitGridOffset; + xTable.gridTable[xBytePos] = xMask | xTable.gridTable[xBytePos]; + + usize yBytePos = yPos + relativeGridBytePos; + uint8 yMask = 1; + yMask <<= bitGridOffset; + yTable.gridTable[yBytePos] = yMask | yTable.gridTable[yBytePos]; + } + } + } +}; + +void SearchTablePositions(std::vector& outputGridMask, usize searchSpace, usize targetPosition, const GridBitMap& selectedTable) +{ + std::vector tempGridMask(selectedTable.rowLength, 0); + + usize xStart = (targetPosition < searchSpace) ? 0 : targetPosition - searchSpace; + usize xEnd = (targetPosition + searchSpace < selectedTable.numPositions) ? targetPosition + searchSpace + 1 : selectedTable.numPositions; + + for(usize pos = xStart; pos < xEnd; pos++) + { + for(usize i = 0; i < selectedTable.rowLength; i++) + { + tempGridMask[i] = tempGridMask[i] | selectedTable.gridTable[(pos * selectedTable.rowLength) + i]; + } + } + + for(usize i = 0; i < selectedTable.rowLength; i++) + { + outputGridMask[i] = tempGridMask[i] & outputGridMask[i]; + } +} + +template +concept IsHGBP = std::is_base_of_v; + +template +std::vector NeighborGridQuery(usize targetGridId, const HGBPT& hyperGridBitMap) +{ + usize searchSpace = std::ceil(std::sqrt(HGBPT::Dimensions)); + + std::vector neighborGridIds = {}; + + std::vector finalGridMask(hyperGridBitMap.xTable.rowLength, std::numeric_limits::max()); + + usize relativeGridBytePos = targetGridId / 8; + uint8 bitGridOffset = targetGridId % 8; + + usize xPos = 0; + for(usize i = 0; i < hyperGridBitMap.xTable.numPositions; i++) + { + usize gridPos = (i * hyperGridBitMap.xTable.rowLength) + relativeGridBytePos; + uint8 mask = 1; + mask <<= bitGridOffset; + uint8 result = hyperGridBitMap.xTable.gridTable[gridPos] & mask; + if(result > 0) + { + xPos = i; + break; + } + } + SearchTablePositions(finalGridMask, searchSpace, xPos, hyperGridBitMap.xTable); + + usize yPos = 0; + for(usize i = 0; i < hyperGridBitMap.yTable.numPositions; i++) + { + usize gridPos = (i * hyperGridBitMap.yTable.rowLength) + relativeGridBytePos; + uint8 mask = 1; + mask <<= bitGridOffset; + uint8 result = hyperGridBitMap.yTable.gridTable[gridPos] & mask; + if(result > 0) + { + yPos = i; + break; + } + } + SearchTablePositions(finalGridMask, searchSpace, yPos, hyperGridBitMap.yTable); + + if constexpr(HGBPT::Dimensions == 3) + { + usize zPos = 0; + for(usize i = 0; i < hyperGridBitMap.zTable.numPositions; i++) + { + usize gridPos = (i * hyperGridBitMap.zTable.rowLength) + relativeGridBytePos; + uint8 mask = 1; + mask <<= bitGridOffset; + uint8 result = hyperGridBitMap.zTable.gridTable[gridPos] & mask; + if(result > 0) + { + zPos = i; + break; + } + } + SearchTablePositions(finalGridMask, searchSpace, zPos, hyperGridBitMap.zTable); + } + + for(usize i = 0; i < finalGridMask.size(); i++) + { + if(finalGridMask[i] > 0) + { + for(uint8 bit = 0; bit < 8; bit++) + { + if((finalGridMask[i] & (1 << bit)) != 0) + { + neighborGridIds.push_back((i * 8) + bit); + } + } + } + } + + return neighborGridIds; +} + +struct ClusterNode +{ + int32 clusterId = 0; + usize parent = 0; +}; + +struct ClusterForest +{ + std::vector clusterForestNodes = {}; + + void initialize(usize numGrids) + { + clusterForestNodes.resize(numGrids); + + for(usize i = 0; i < clusterForestNodes.size(); i++) + { + clusterForestNodes[i].parent = i; + clusterForestNodes[i].clusterId = static_cast(i + 1); + } + } + + usize findClusterRoot(usize gridId) + { + if(clusterForestNodes[gridId].parent == gridId) + { + return gridId; + } + + return findClusterRoot(clusterForestNodes[gridId].parent); + } + + bool infer(usize pGridId, usize qGridId) + { + return findClusterRoot(pGridId) == findClusterRoot(qGridId); + } + + void mergeLRC(const std::vector& gridIds) + { + if(gridIds.size() < 2) + { + return; + } + + std::vector rootClusterIdx = {}; + + usize lowestClusterIdx = findClusterRoot(gridIds[0]); + rootClusterIdx.push_back(lowestClusterIdx); + for(usize i = 1; i < gridIds.size(); i++) + { + usize clusterIndex = findClusterRoot(gridIds[i]); + rootClusterIdx.push_back(clusterIndex); + if(clusterForestNodes[clusterIndex].clusterId < clusterForestNodes[lowestClusterIdx].clusterId) + { + lowestClusterIdx = clusterIndex; + } + } + + for(const usize clusterIdx : rootClusterIdx) + { + if(lowestClusterIdx != clusterIdx) + { + clusterForestNodes[clusterIdx].parent = clusterForestNodes[lowestClusterIdx].parent; + } + } + } +}; + +/** + * @brief OOC GDCF: uses on-demand per-grid-cell reads for canMerge. + */ +template +class GDCF +{ +public: + GDCF() = delete; + GDCF(const std::atomic_bool& shouldCancel, const AbstractDataStore& inputArray, float32 epsilon, const std::unique_ptr& mask, + ClusterUtilities::DistanceMetric distMetric) + : hyperGridBitMap(HGBPT(shouldCancel, inputArray, epsilon, mask)) + , m_Epsilon(epsilon) + , m_InputDataStore(inputArray) + , m_DistMetric(distMetric) + , m_ShouldCancel(shouldCancel) + { + } + + Result<> cluster(usize minPoints, DBSCAN::ParseOrder parseOrder, std::mt19937_64::result_type seed = std::mt19937_64::default_seed) + { + std::vector coreGridIds = {}; + for(usize i = 0; i < hyperGridBitMap.gridVoxels.size(); i++) + { + if(hyperGridBitMap.gridVoxels[i].size() >= minPoints) + { + coreGridIds.push_back(i); + } + } + if(coreGridIds.empty()) + { + return MakeWarningVoidResult(-85640, "No clusters detected - Consider reducing number of required points (`Minimum Points`) or increasing acceptable distance (`Epsilon`)."); + } + + if(m_ShouldCancel) + { + return {}; + } + + switch(parseOrder) + { + case DBSCAN::ParseOrder::LowDensityFirst: { + QuickSortGrids(coreGridIds, 0, coreGridIds.size() - 1); + break; + } + case DBSCAN::ParseOrder::Random: { + std::mt19937_64 gen(seed); + std::uniform_real_distribution dist(0, 1); + + auto maxIdx = static_cast(coreGridIds.size() - 1); + + for(usize i = 1; i < coreGridIds.size(); i++) + { + auto r = static_cast(std::floor(dist(gen) * maxIdx)); + std::swap(coreGridIds[i], coreGridIds[r]); + } + + break; + } + case DBSCAN::SeededRandom: { + std::mt19937_64 gen(seed); + std::uniform_real_distribution dist(0, 1); + + auto maxIdx = static_cast(coreGridIds.size() - 1); + + for(usize i = 1; i < coreGridIds.size(); i++) + { + auto r = static_cast(std::floor(dist(gen) * maxIdx)); + std::swap(coreGridIds[i], coreGridIds[r]); + } + + break; + } + } + + if(m_ShouldCancel) + { + return {}; + } + + clusterForest.initialize(hyperGridBitMap.gridVoxels.size()); + for(usize i = 0; i < coreGridIds.size(); i++) + { + if(m_ShouldCancel) + { + return {}; + } + + std::vector neighborGrids = NeighborGridQuery(coreGridIds[i], hyperGridBitMap); + + std::vector cluster = {}; + cluster.push_back(coreGridIds[i]); + for(const usize gridId : neighborGrids) + { + if(clusterForest.infer(coreGridIds[i], gridId)) + { + continue; + } + + if(canMerge(coreGridIds[i], gridId)) + { + if(hyperGridBitMap.gridVoxels[gridId].size() < minPoints && clusterForest.clusterForestNodes[gridId].parent == gridId) + { + clusterForest.clusterForestNodes[gridId].parent = coreGridIds[i]; + } + else + { + cluster.push_back(gridId); + } + } + } + + clusterForest.mergeLRC(cluster); + } + + // Now determine if non-core grids are close enough to a cluster to be border else noise + usize operations = 0; + do + { + operations = 0; + for(usize i = 0; i < hyperGridBitMap.gridVoxels.size(); i++) + { + if(m_ShouldCancel) + { + return {}; + } + + if(hyperGridBitMap.gridVoxels[i].size() < minPoints) + { + std::vector neighborGrids = NeighborGridQuery(i, hyperGridBitMap); + + for(const usize gridId : neighborGrids) + { + if(clusterForest.infer(i, gridId)) + { + continue; + } + + if(canMerge(i, gridId)) + { + usize activeParent = clusterForest.findClusterRoot(i); + usize neighborGridParent = clusterForest.findClusterRoot(gridId); + if(activeParent == i) + { + if(hyperGridBitMap.gridVoxels[gridId].size() < minPoints && neighborGridParent == gridId) + { + continue; + } + clusterForest.clusterForestNodes[i].parent = neighborGridParent; + } + else + { + if(hyperGridBitMap.gridVoxels[gridId].size() < minPoints && neighborGridParent == gridId) + { + clusterForest.clusterForestNodes[gridId].parent = activeParent; + } + else + { + if(clusterForest.clusterForestNodes[activeParent].clusterId < clusterForest.clusterForestNodes[neighborGridParent].clusterId) + { + clusterForest.clusterForestNodes[neighborGridParent].parent = activeParent; + } + else + { + clusterForest.clusterForestNodes[activeParent].parent = neighborGridParent; + } + } + } + operations++; + } + } + } + } + } while(operations > 0); + + std::vector clusters = {}; + for(usize i = 0; i < clusterForest.clusterForestNodes.size(); i++) + { + if(clusterForest.clusterForestNodes[i].parent == i) + { + if(hyperGridBitMap.gridVoxels[i].size() >= minPoints) + { + clusters.push_back(i); + } + else + { + clusterForest.clusterForestNodes[i].clusterId = 0; + } + } + } + + for(usize i = 0; i < clusters.size(); i++) + { + clusterForest.clusterForestNodes[clusters[i]].clusterId = static_cast(i + 1); + } + + return {}; + } + + Result<> label(AbstractDataStore& fIdsDataStore) + { + if(clusterForest.clusterForestNodes.empty()) + { + return MakeWarningVoidResult(-85640, "No clusters detected - Consider reducing number of required points (`Minimum Points`) or increasing acceptable distance (`Epsilon`)."); + } + + fIdsDataStore.fill(0); + for(usize gridIdx = 0; gridIdx < hyperGridBitMap.gridVoxels.size(); gridIdx++) + { + if(m_ShouldCancel) + { + return {}; + } + + int32 featureId = clusterForest.clusterForestNodes[clusterForest.findClusterRoot(gridIdx)].clusterId; + for(usize pointIdx : hyperGridBitMap.gridVoxels[gridIdx]) + { + fIdsDataStore.setValue(pointIdx, featureId); + } + } + + return {}; + } + +private: + HGBPT hyperGridBitMap; + + ClusterForest clusterForest = {}; + + float32 m_Epsilon = 0.0f; + const AbstractDataStore& m_InputDataStore; + ClusterUtilities::DistanceMetric m_DistMetric; + const std::atomic_bool& m_ShouldCancel; + + /** + * @brief Reads coordinate data for all points in a grid cell from the store. + * Memory cost is O(gridCellSize * dims), not O(n). + */ + std::vector readGridCellCoords(usize gridId) const + { + const auto& indices = hyperGridBitMap.gridVoxels[gridId]; + const usize dims = static_cast(HGBPT::Dimensions); + std::vector coords(indices.size() * dims); + auto tupleBuf = std::make_unique(dims); + for(usize i = 0; i < indices.size(); i++) + { + m_InputDataStore.copyIntoBuffer(indices[i] * dims, nonstd::span(tupleBuf.get(), dims)); + for(usize d = 0; d < dims; d++) + { + coords[i * dims + d] = static_cast(tupleBuf[d]); + } + } + return coords; + } + + // Uses Hoare's method for speed + usize ProcessSection(std::vector& sorted, usize begin, usize end) const + { + const usize threshold = hyperGridBitMap.gridVoxels[sorted[begin]].size(); + + usize front = begin; + usize back = end; + + while(true) + { + while(hyperGridBitMap.gridVoxels[sorted[front]].size() < threshold) + { + front++; + } + + while(hyperGridBitMap.gridVoxels[sorted[back]].size() > threshold) + { + back--; + } + + if(front >= back) + { + return back; + } + + std::swap(sorted[front], sorted[back]); + front++; + back--; + } + } + + void QuickSortGrids(std::vector& sorted, usize begin, usize end) const + { + if(begin >= end) + { + return; + } + + usize next = ProcessSection(sorted, begin, end); + + QuickSortGrids(sorted, begin, next); + QuickSortGrids(sorted, next + 1, end); + } + + // OOC path: read grid cell coords on-demand into O(gridCellSize) local buffers + bool canMerge(usize pGridId, usize qGridId) + { + const usize dims = static_cast(HGBPT::Dimensions); + auto pCoords = readGridCellCoords(pGridId); + auto qCoords = readGridCellCoords(qGridId); + + for(usize p = 0; p < hyperGridBitMap.gridVoxels[pGridId].size(); p++) + { + for(usize q = 0; q < hyperGridBitMap.gridVoxels[qGridId].size(); q++) + { + float64 dist = ClusterUtilities::GetDistance(pCoords, dims * p, qCoords, dims * q, dims, m_DistMetric); + if(dist < m_Epsilon) + { + return true; + } + } + } + return false; + } +}; + +template +Result<> RunAlgorithm(const DBSCANInputValues* inputValues, const AbstractDataStore& inputArray, const std::unique_ptr& mask, Int32Array& featureIds, + const std::atomic_bool& shouldCancel) +{ + AlgorithmT algorithm = AlgorithmT(shouldCancel, inputArray, inputValues->Epsilon, mask, inputValues->DistanceMetric); + + if(shouldCancel) + { + return {}; + } + + Result<> result = algorithm.cluster(inputValues->MinPoints, static_cast(inputValues->ParseOrder), inputValues->Seed); + if(result.invalid() || !result.warnings().empty()) + { + return result; + } + + if(shouldCancel) + { + return {}; + } + + return algorithm.label(featureIds.getDataStoreRef()); +} + +struct DBSCANScanlineFunctor +{ + template + Result<> operator()(const DBSCANInputValues* inputValues, const IDataArray& clusterArray, const std::unique_ptr& mask, Int32Array& featureIds, + const std::atomic_bool& shouldCancel) + { + const auto& inputArray = dynamic_cast&>(clusterArray).getDataStoreRef(); + if(inputArray.getNumberOfComponents() == 2) + { + return RunAlgorithm, T>(inputValues, inputArray, mask, featureIds, shouldCancel); + } + else if(inputArray.getNumberOfComponents() == 3) + { + return RunAlgorithm, T>(inputValues, inputArray, mask, featureIds, shouldCancel); + } + else + { + return MakeErrorResult(-54060, "Input components invalid. Only 2 or 3 accepted."); + } + + return {}; + } +}; +} // namespace + +// ----------------------------------------------------------------------------- +DBSCANScanline::DBSCANScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const DBSCANInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +DBSCANScanline::~DBSCANScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +/** + * @brief OOC DBSCAN execution using chunked copyIntoBuffer I/O. + */ +Result<> DBSCANScanline::operator()() +{ + auto& clusteringArray = m_DataStructure.getDataRefAs(m_InputValues->ClusteringArrayPath); + auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); + + std::unique_ptr maskCompare; + try + { + maskCompare = MaskCompareUtilities::InstantiateMaskCompare(m_DataStructure, m_InputValues->MaskArrayPath); + } catch(const std::out_of_range& exception) + { + std::string message = fmt::format("Mask Array DataPath does not exist or is not of the correct type (Bool | UInt8) {}", m_InputValues->MaskArrayPath.toString()); + return MakeErrorResult(-54060, message); + } + + Result<> result = ExecuteDataFunction(DBSCANScanlineFunctor{}, clusteringArray.getDataType(), m_InputValues, clusteringArray, maskCompare, featureIds, m_ShouldCancel); + if(result.invalid()) + { + return result; + } + + if(m_ShouldCancel) + { + return {}; + } + + auto& featureIdsDataStore = featureIds.getDataStoreRef(); + int32 maxCluster = 0; + { + const usize totalSize = featureIdsDataStore.getSize(); + constexpr usize k_ChunkSize = 1000000; + std::vector maxBuf(std::min(totalSize, k_ChunkSize)); + for(usize start = 0; start < totalSize; start += k_ChunkSize) + { + usize count = std::min(k_ChunkSize, totalSize - start); + featureIdsDataStore.copyIntoBuffer(start, nonstd::span(maxBuf.data(), count)); + for(usize i = 0; i < count; i++) + { + maxCluster = std::max(maxCluster, maxBuf[i]); + } + } + } + m_DataStructure.getDataAs(m_InputValues->FeatureAM)->resizeTuples(ShapeType{static_cast(maxCluster + 1)}); + + return result; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp new file mode 100644 index 0000000000..b4da78384d --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp @@ -0,0 +1,39 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct DBSCANInputValues; + +/** + * @class DBSCANScanline + * @brief Out-of-core algorithm for DBSCAN. Uses chunked copyIntoBuffer bulk I/O + * for grid construction and on-demand per-grid-cell reads for canMerge distance + * computation. Selected by DispatchAlgorithm when any input array is backed by + * ZarrStore (out-of-core storage). + */ +class SIMPLNXCORE_EXPORT DBSCANScanline +{ +public: + DBSCANScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const DBSCANInputValues* inputValues); + ~DBSCANScanline() noexcept; + + DBSCANScanline(const DBSCANScanline&) = delete; + DBSCANScanline(DBSCANScanline&&) noexcept = delete; + DBSCANScanline& operator=(const DBSCANScanline&) = delete; + DBSCANScanline& operator=(DBSCANScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const DBSCANInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp index 045546d49e..c3a0014387 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp @@ -4,66 +4,10 @@ #include "simplnx/DataStructure/DataGroup.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/DataGroupUtilities.hpp" -#include "simplnx/Utilities/MessageHelper.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" -#include "simplnx/Utilities/ParallelTaskAlgorithm.hpp" +#include "simplnx/Utilities/SliceBufferedTransfer.hpp" using namespace nx::core; -namespace -{ -class ErodeDilateBadDataTransferDataImpl -{ -public: - ErodeDilateBadDataTransferDataImpl() = delete; - ErodeDilateBadDataTransferDataImpl(const ErodeDilateBadDataTransferDataImpl&) = default; - - ErodeDilateBadDataTransferDataImpl(ErodeDilateBadData* filterAlg, usize totalPoints, ChoicesParameter::ValueType operation, const Int32AbstractDataStore& featureIds, - const std::vector& neighbors, const std::shared_ptr& dataArrayPtr, MessageHelper& messageHelper) - : m_FilterAlg(filterAlg) - , m_TotalPoints(totalPoints) - , m_Operation(operation) - , m_Neighbors(neighbors) - , m_DataArrayPtr(dataArrayPtr) - , m_FeatureIds(featureIds) - , m_MessageHelper(messageHelper) - { - } - ErodeDilateBadDataTransferDataImpl(ErodeDilateBadDataTransferDataImpl&&) = default; // Move Constructor Not Implemented - ErodeDilateBadDataTransferDataImpl& operator=(const ErodeDilateBadDataTransferDataImpl&) = delete; // Copy Assignment Not Implemented - ErodeDilateBadDataTransferDataImpl& operator=(ErodeDilateBadDataTransferDataImpl&&) = delete; // Move Assignment Not Implemented - - ~ErodeDilateBadDataTransferDataImpl() = default; - - void operator()() const - { - ThrottledMessenger throttledMessenger = m_MessageHelper.createThrottledMessenger(); - std::string arrayName = m_DataArrayPtr->getName(); - for(usize i = 0; i < m_TotalPoints; i++) - { - throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Processing {}: {:.2f}% completed", arrayName, CalculatePercentComplete(i, m_TotalPoints)); }); - - const int32 featureName = m_FeatureIds[i]; - const int64 neighbor = m_Neighbors[i]; - if(neighbor >= 0) - { - if((featureName == 0 && m_FeatureIds[neighbor] > 0 && m_Operation == detail::k_ErodeIndex) || (featureName > 0 && m_FeatureIds[neighbor] == 0 && m_Operation == detail::k_DilateIndex)) - { - m_DataArrayPtr->copyTuple(neighbor, i); - } - } - } - } - -private: - ErodeDilateBadData* m_FilterAlg = nullptr; - usize m_TotalPoints = 0; - ChoicesParameter::ValueType m_Operation = 0; - std::vector m_Neighbors; - const std::shared_ptr m_DataArrayPtr; - const Int32AbstractDataStore& m_FeatureIds; - MessageHelper& m_MessageHelper; -}; -} // namespace // ----------------------------------------------------------------------------- ErodeDilateBadData::ErodeDilateBadData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ErodeDilateBadDataInputValues* inputValues) @@ -78,7 +22,7 @@ ErodeDilateBadData::ErodeDilateBadData(DataStructure& dataStructure, const IFilt ErodeDilateBadData::~ErodeDilateBadData() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ErodeDilateBadData::getCancel() +const std::atomic_bool& ErodeDilateBadData::getCancel() const { return m_ShouldCancel; } @@ -86,53 +30,132 @@ const std::atomic_bool& ErodeDilateBadData::getCancel() // ----------------------------------------------------------------------------- Result<> ErodeDilateBadData::operator()() { - const auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); - const usize totalPoints = featureIds.getNumberOfTuples(); - - std::vector neighbors(totalPoints, -1); + const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath).getDataStoreRef(); const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometry); - SizeVec3 udims = selectedImageGeom.getDimensions(); + std::array dims = {static_cast(udims[0]), static_cast(udims[1]), static_cast(udims[2])}; - std::array dims = { - static_cast(udims[0]), - static_cast(udims[1]), - static_cast(udims[2]), - }; + std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); + std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + + // Find max feature ID using Z-slice batched reads usize numFeatures = 0; - for(usize i = 0; i < totalPoints; i++) { - const int32 featureName = featureIds[i]; - if(featureName > numFeatures) + std::vector sliceBuf(sliceSize); + for(int64 z = 0; z < dims[2]; z++) { - numFeatures = featureName; + featureIds.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(sliceBuf.data(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(sliceBuf[i] > static_cast(numFeatures)) + { + numFeatures = sliceBuf[i]; + } + } } } - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - std::vector featureCount(numFeatures + 1, 0); + // FeatureIds rolling window for neighbor lookups + std::array, 3> featureIdSlices; + for(auto& fis : featureIdSlices) + { + fis.resize(sliceSize); + } + + auto readFeatureIdSlice = [&](int64 z, usize slot) { featureIds.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(featureIdSlices[slot].data(), sliceSize)); }; + + constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + + // Per-slice mark arrays: marks[0]=z-1, marks[1]=z, marks[2]=z+1 + // Each entry is -1 (no transfer) or the global source index. + // This replaces the O(totalPoints) neighbors array with O(3*sliceSize). + std::array, 3> marks; + for(auto& m : marks) + { + m.resize(sliceSize); + } + + const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsArrayPath, m_InputValues->IgnoredDataArrayPaths); + const usize dimZ = static_cast(dims[2]); + + // Helper to transfer a single Z-slice across all arrays + auto transferSlice = [&](usize z, const std::vector& sliceMarks) { + for(const auto& voxelArray : voxelArrays) + { + SliceBufferedTransferOneZ(*voxelArray, sliceMarks, sliceSize, z, dimZ); + } + }; + for(int32 iteration = 0; iteration < m_InputValues->NumIterations; iteration++) { + // Clear marks + for(auto& m : marks) + { + std::fill(m.begin(), m.end(), -1); + } + + // Initialize FeatureId rolling window + readFeatureIdSlice(0, 1); + if(dims[2] > 1) + { + readFeatureIdSlice(1, 2); + } + for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - const int64 zStride = dims[0] * dims[1] * zIdx; + // Advance FeatureId rolling window + if(zIdx > 0) + { + std::swap(featureIdSlices[0], featureIdSlices[1]); + std::swap(featureIdSlices[1], featureIdSlices[2]); + if(zIdx + 1 < dims[2]) + { + readFeatureIdSlice(zIdx + 1, 2); + } + } + + // Find neighbors for bad voxels in this Z-slice for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { - const int64 yStride = dims[0] * yIdx; for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { - const int64 voxelIndex = zStride + yStride + xIdx; - const int32 featureName = featureIds[voxelIndex]; + const usize inSlice = static_cast(yIdx * dims[0] + xIdx); + const int32 featureName = featureIdSlices[1][inSlice]; if(featureName == 0) { int32 most = 0; - // Loop over the 6 face neighbors of the voxel std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + if(!m_InputValues->XDirOn) + { + isValidFaceNeighbor[k_NegativeXNeighbor] = false; + isValidFaceNeighbor[k_PositiveXNeighbor] = false; + } + if(!m_InputValues->YDirOn) + { + isValidFaceNeighbor[k_NegativeYNeighbor] = false; + isValidFaceNeighbor[k_PositiveYNeighbor] = false; + } + if(!m_InputValues->ZDirOn) + { + isValidFaceNeighbor[k_NegativeZNeighbor] = false; + isValidFaceNeighbor[k_PositiveZNeighbor] = false; + } + + const int64 voxelIndex = xIdx + yIdx * dims[0] + zIdx * static_cast(sliceSize); + const std::array neighborInSlice = { + inSlice, // -Z + static_cast((yIdx - 1) * dims[0] + xIdx), // -Y + static_cast(yIdx * dims[0] + (xIdx - 1)), // -X + static_cast(yIdx * dims[0] + (xIdx + 1)), // +X + static_cast((yIdx + 1) * dims[0] + xIdx), // +Y + inSlice // +Z + }; + for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) @@ -140,11 +163,13 @@ Result<> ErodeDilateBadData::operator()() continue; } const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + const int32 feature = featureIdSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; - const int32 feature = featureIds[neighborPoint]; if(m_InputValues->Operation == detail::k_DilateIndex && feature > 0) { - neighbors[neighborPoint] = voxelIndex; + // Mark the good NEIGHBOR to be overwritten by this bad voxel. + // The neighbor is in slot k_NeighborSlot[faceIndex] (0=z-1, 1=z, 2=z+1). + marks[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] = voxelIndex; } if(feature > 0 && m_InputValues->Operation == detail::k_ErodeIndex) { @@ -153,55 +178,44 @@ Result<> ErodeDilateBadData::operator()() if(current > most) { most = current; - neighbors[voxelIndex] = neighborPoint; + // Mark this bad voxel to be overwritten by the best neighbor + marks[1][inSlice] = neighborPoint; } } } if(m_InputValues->Operation == detail::k_ErodeIndex) { - // Loop over the 6 face neighbors of the voxel - isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) { continue; } - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - - const int32 feature = featureIds[neighborPoint]; + const int32 feature = featureIdSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; featureCount[feature] = 0; } } } } } - } - - // Build up a list of the DataArrays that we are going to operate on. - const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsArrayPath, m_InputValues->IgnoredDataArrayPaths); - - MessageHelper messageHelper(m_MessageHandler); - ParallelTaskAlgorithm taskRunner; - taskRunner.setParallelizationEnabled(true); - for(const auto& voxelArray : voxelArrays) - { - // We need to skip updating the FeatureIds until all the other arrays are updated - // since we actually depend on the feature Ids values. - if(voxelArray->getName() == m_InputValues->FeatureIdsArrayPath.getTargetName()) + // Transfer z-1: all marks for z-1 are now complete (from bad voxels at z-2, z-1, z) + if(zIdx > 0) { - continue; + transferSlice(static_cast(zIdx - 1), marks[0]); } - taskRunner.execute(ErodeDilateBadDataTransferDataImpl(this, totalPoints, m_InputValues->Operation, featureIds, neighbors, voxelArray, messageHelper)); + // Rotate marks: [0]=old[1], [1]=old[2], [2]=cleared + std::swap(marks[0], marks[1]); + std::swap(marks[1], marks[2]); + std::fill(marks[2].begin(), marks[2].end(), -1); } - taskRunner.wait(); // This will spill over if the number of DataArrays to process does not divide evenly by the number of threads. - // Now update the feature Ids - auto featureIDataArray = m_DataStructure.getSharedDataAs(m_InputValues->FeatureIdsArrayPath); - taskRunner.setParallelizationEnabled(false); // Do this to make the next call synchronous - taskRunner.execute(ErodeDilateBadDataTransferDataImpl(this, totalPoints, m_InputValues->Operation, featureIds, neighbors, featureIDataArray, messageHelper)); + // Transfer last slice + if(dims[2] > 0) + { + transferSlice(static_cast(dims[2] - 1), marks[0]); + } } return {}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp index 5eb5a734b4..10ccc8d537 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp @@ -20,6 +20,10 @@ static inline constexpr ChoicesParameter::ValueType k_DilateIndex = 0ULL; static inline constexpr ChoicesParameter::ValueType k_ErodeIndex = 1ULL; } // namespace detail +/** + * @struct ErodeDilateBadDataInputValues + * @brief Holds all user-supplied parameters for the ErodeDilateBadData algorithm. + */ struct SIMPLNXCORE_EXPORT ErodeDilateBadDataInputValues { ChoicesParameter::ValueType Operation; @@ -33,13 +37,24 @@ struct SIMPLNXCORE_EXPORT ErodeDilateBadDataInputValues }; /** - * @class ConditionalSetValueFilter - + * @class ErodeDilateBadData + * @brief Erodes or dilates bad (FeatureId == 0) voxels by replacing them with the most common neighbor feature. */ class SIMPLNXCORE_EXPORT ErodeDilateBadData { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays + * @param mesgHandler Handler for sending progress messages to the UI + * @param shouldCancel Atomic flag checked between iterations to support cancellation + * @param inputValues User-supplied parameters controlling the algorithm behavior + */ ErodeDilateBadData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ErodeDilateBadDataInputValues* inputValues); + + /** + * @brief Default destructor. + */ ~ErodeDilateBadData() noexcept; ErodeDilateBadData(const ErodeDilateBadData&) = delete; @@ -47,9 +62,13 @@ class SIMPLNXCORE_EXPORT ErodeDilateBadData ErodeDilateBadData& operator=(const ErodeDilateBadData&) = delete; ErodeDilateBadData& operator=(ErodeDilateBadData&&) noexcept = delete; + /** + * @brief Executes the erode/dilate bad data algorithm. + * @return Result<> indicating success or any errors encountered during execution + */ Result<> operator()(); - const std::atomic_bool& getCancel(); + const std::atomic_bool& getCancel() const; private: DataStructure& m_DataStructure; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp index ffb9123d1d..7f48fd1b8b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp @@ -5,21 +5,9 @@ #include "simplnx/Utilities/DataGroupUtilities.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" +#include "simplnx/Utilities/SliceBufferedTransfer.hpp" using namespace nx::core; -namespace -{ -struct DataArrayCopyTupleFunctor -{ - template - void operator()(IDataArray& outputIDataArray, size_t sourceIndex, size_t targetIndex) - { - using DataArrayType = DataArray; - DataArrayType outputArray = dynamic_cast(outputIDataArray); - outputArray.copyTuple(sourceIndex, targetIndex); - } -}; -} // namespace // ----------------------------------------------------------------------------- ErodeDilateCoordinationNumber::ErodeDilateCoordinationNumber(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, @@ -35,7 +23,7 @@ ErodeDilateCoordinationNumber::ErodeDilateCoordinationNumber(DataStructure& data ErodeDilateCoordinationNumber::~ErodeDilateCoordinationNumber() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ErodeDilateCoordinationNumber::getCancel() +const std::atomic_bool& ErodeDilateCoordinationNumber::getCancel() const { return m_ShouldCancel; } @@ -43,44 +31,86 @@ const std::atomic_bool& ErodeDilateCoordinationNumber::getCancel() // ----------------------------------------------------------------------------- Result<> ErodeDilateCoordinationNumber::operator()() { - const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); - const size_t totalPoints = featureIds.getNumberOfTuples(); - - std::vector neighbors(totalPoints, -1); const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometry); - SizeVec3 udims = selectedImageGeom.getDimensions(); + std::array dims = {static_cast(udims[0]), static_cast(udims[1]), static_cast(udims[2])}; - std::array dims = { - static_cast(udims[0]), - static_cast(udims[1]), - static_cast(udims[2]), - }; + std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); + std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - size_t numFeatures = 0; + const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsArrayPath, m_InputValues->IgnoredDataArrayPaths); + + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + const usize dimZ = static_cast(dims[2]); - for(size_t i = 0; i < totalPoints; i++) + // Find max feature ID using Z-slice batched reads + const auto& featureIdsStore = featureIds.getDataStoreRef(); + usize numFeatures = 0; { - const int32 featureName = featureIds[i]; - if(featureName > numFeatures) + std::vector sliceBuf(sliceSize); + for(int64 z = 0; z < dims[2]; z++) { - numFeatures = featureName; + featureIdsStore.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(sliceBuf.data(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(sliceBuf[i] > static_cast(numFeatures)) + { + numFeatures = sliceBuf[i]; + } + } } } - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - - const std::string attrMatName = m_InputValues->FeatureIdsArrayPath.getTargetName(); - const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsArrayPath, m_InputValues->IgnoredDataArrayPaths); - std::vector featureCount(numFeatures + 1, 0); - std::vector coordinationNumber(totalPoints, 0); bool keepGoing = true; int32 counter = 1; + // FeatureIds rolling window + std::array, 3> featureIdSlices; + for(auto& fis : featureIdSlices) + { + fis.resize(sliceSize); + } + + auto readFeatureIdSlice = [&](int64 z, usize slot) { featureIdsStore.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(featureIdSlices[slot].data(), sliceSize)); }; + + constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + + // Per-slice neighbors (O(3*sliceSize) replaces O(totalPoints) neighbors array) + // Slot 0=z-1, slot 1=z, slot 2=z+1 + std::array, 3> sliceNeighbors; + for(auto& sn : sliceNeighbors) + { + sn.resize(sliceSize, -1); + } + + // Per-slice coordination numbers (O(3*sliceSize) replaces O(totalPoints)) + std::array, 3> sliceCoordination; + for(auto& sc : sliceCoordination) + { + sc.resize(sliceSize, 0); + } + + // Helper to transfer a single Z-slice across all arrays, for qualifying voxels + auto transferSlice = [&](usize z, const std::vector& marks, const std::vector& coord) { + // Filter marks: only transfer voxels meeting coordination threshold + std::vector filteredMarks(sliceSize, -1); + for(usize i = 0; i < sliceSize; i++) + { + if(coord[i] >= m_InputValues->CoordinationNumber && coord[i] > 0) + { + filteredMarks[i] = marks[i]; + counter++; + } + } + for(const auto& voxelArray : voxelArrays) + { + SliceBufferedTransferOneZ(*voxelArray, filteredMarks, sliceSize, z, dimZ); + } + }; + while(counter > 0 && keepGoing) { counter = 0; @@ -89,20 +119,56 @@ Result<> ErodeDilateCoordinationNumber::operator()() keepGoing = false; } + // Clear per-slice arrays + for(auto& sn : sliceNeighbors) + { + std::fill(sn.begin(), sn.end(), -1); + } + for(auto& sc : sliceCoordination) + { + std::fill(sc.begin(), sc.end(), 0); + } + + // Initialize rolling window + readFeatureIdSlice(0, 1); + if(dims[2] > 1) + { + readFeatureIdSlice(1, 2); + } + for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - const int64 zStride = dims[0] * dims[1] * zIdx; + if(zIdx > 0) + { + std::swap(featureIdSlices[0], featureIdSlices[1]); + std::swap(featureIdSlices[1], featureIdSlices[2]); + if(zIdx + 1 < dims[2]) + { + readFeatureIdSlice(zIdx + 1, 2); + } + } + for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { - const int64 yStride = dims[0] * yIdx; for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { - const int64 voxelIndex = zStride + yStride + xIdx; - const int32 featureName = featureIds[voxelIndex]; + const int64 voxelIndex = dims[0] * dims[1] * zIdx + dims[0] * yIdx + xIdx; + const usize inSlice = static_cast(yIdx * dims[0] + xIdx); + const int32 featureName = featureIdSlices[1][inSlice]; int32 coordination = 0; int32 most = 0; - // Loop over the 6 face neighbors of the voxel + std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + + const std::array neighborInSlice = { + inSlice, // -Z + static_cast((yIdx - 1) * dims[0] + xIdx), // -Y + static_cast(yIdx * dims[0] + (xIdx - 1)), // -X + static_cast(yIdx * dims[0] + (xIdx + 1)), // +X + static_cast((yIdx + 1) * dims[0] + xIdx), // +Y + inSlice // +Z + }; + for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) @@ -111,8 +177,8 @@ Result<> ErodeDilateCoordinationNumber::operator()() } const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + const int32 feature = featureIdSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; - const int32 feature = featureIds[neighborPoint]; if((featureName > 0 && feature == 0) || (featureName == 0 && feature > 0)) { coordination = coordination + 1; @@ -121,36 +187,20 @@ Result<> ErodeDilateCoordinationNumber::operator()() if(current > most) { most = current; - neighbors[voxelIndex] = neighborPoint; + sliceNeighbors[1][inSlice] = neighborPoint; } } } - coordinationNumber[voxelIndex] = coordination; - const int64 neighbor = neighbors[voxelIndex]; - if(coordinationNumber[voxelIndex] >= m_InputValues->CoordinationNumber && coordinationNumber[voxelIndex] > 0) - { - // TODO: update to use IDataArray->copyTuple() function - /****************************************************************** - * If this section is slow it is because we are having to use the - * ExecuteDataFunction() in order to call "copyTuple()" because - * "copyTuple()" isn't in the IArray API set. Oh well. - */ - for(const auto& voxelArray : voxelArrays) - { - ExecuteDataFunction(DataArrayCopyTupleFunctor{}, voxelArray->getDataType(), *voxelArray, neighbor, voxelIndex); - } - } - // Loop over the 6 face neighbors of the voxel - isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + sliceCoordination[1][inSlice] = coordination; + + // Reset featureCount for neighbors for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) { continue; } - - const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - const int32 feature = featureIds[neighborPoint]; + const int32 feature = featureIdSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; if(feature > 0) { featureCount[feature] = 0; @@ -158,22 +208,27 @@ Result<> ErodeDilateCoordinationNumber::operator()() } } } - } - for(int64 zIndex = 0; zIndex < dims[2]; zIndex++) - { - const auto zStride = static_cast(dims[0] * dims[1] * zIndex); - for(int64 yIndex = 0; yIndex < dims[1]; yIndex++) + + // Transfer z-1 (complete after processing z) + if(zIdx > 0) { - const auto yStride = static_cast(dims[0] * yIndex); - for(int64 xIndex = 0; xIndex < dims[0]; xIndex++) - { - const int64 voxelIndex = zStride + yStride + xIndex; - if(coordinationNumber[voxelIndex] >= m_InputValues->CoordinationNumber) - { - counter++; - } - } + transferSlice(static_cast(zIdx - 1), sliceNeighbors[0], sliceCoordination[0]); } + + // Rotate per-slice arrays + std::swap(sliceNeighbors[0], sliceNeighbors[1]); + std::swap(sliceNeighbors[1], sliceNeighbors[2]); + std::fill(sliceNeighbors[2].begin(), sliceNeighbors[2].end(), -1); + + std::swap(sliceCoordination[0], sliceCoordination[1]); + std::swap(sliceCoordination[1], sliceCoordination[2]); + std::fill(sliceCoordination[2].begin(), sliceCoordination[2].end(), 0); + } + + // Transfer last slice + if(dims[2] > 0) + { + transferSlice(static_cast(dims[2] - 1), sliceNeighbors[0], sliceCoordination[0]); } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp index 28eb0fd465..214a165693 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp @@ -13,6 +13,10 @@ namespace nx::core { +/** + * @struct ErodeDilateCoordinationNumberInputValues + * @brief Holds all user-supplied parameters for the ErodeDilateCoordinationNumber algorithm. + */ struct SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumberInputValues { int32 CoordinationNumber; @@ -23,12 +27,24 @@ struct SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumberInputValues }; /** - * @class + * @class ErodeDilateCoordinationNumber + * @brief Smooths voxel boundaries by eroding or dilating based on coordination number thresholds. */ class SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumber { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays + * @param mesgHandler Handler for sending progress messages to the UI + * @param shouldCancel Atomic flag checked between iterations to support cancellation + * @param inputValues User-supplied parameters controlling the algorithm behavior + */ ErodeDilateCoordinationNumber(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ErodeDilateCoordinationNumberInputValues* inputValues); + + /** + * @brief Default destructor. + */ ~ErodeDilateCoordinationNumber() noexcept; ErodeDilateCoordinationNumber(const ErodeDilateCoordinationNumber&) = delete; @@ -36,9 +52,13 @@ class SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumber ErodeDilateCoordinationNumber& operator=(const ErodeDilateCoordinationNumber&) = delete; ErodeDilateCoordinationNumber& operator=(ErodeDilateCoordinationNumber&&) noexcept = delete; + /** + * @brief Executes the erode/dilate coordination number algorithm. + * @return Result<> indicating success or any errors encountered during execution + */ Result<> operator()(); - const std::atomic_bool& getCancel(); + const std::atomic_bool& getCancel() const; private: DataStructure& m_DataStructure; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp index a6ebfd4999..32ed5f7cb8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp @@ -20,7 +20,7 @@ ErodeDilateMask::ErodeDilateMask(DataStructure& dataStructure, const IFilter::Me ErodeDilateMask::~ErodeDilateMask() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ErodeDilateMask::getCancel() +const std::atomic_bool& ErodeDilateMask::getCancel() const { return m_ShouldCancel; } @@ -30,9 +30,7 @@ Result<> ErodeDilateMask::operator()() { auto& mask = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); - const size_t totalPoints = mask.getNumberOfTuples(); - - std::vector maskCopy(totalPoints, false); + const usize totalPoints = mask.getNumberOfTuples(); const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometry); @@ -47,28 +45,100 @@ Result<> ErodeDilateMask::operator()() std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - for(int32_t iteration = 0; iteration < m_InputValues->NumIterations; iteration++) + // Z-slice buffering: maintain rolling window of 3 adjacent Z-slices for mask + // to avoid random OOC chunk access during neighbor lookups. + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + + // Rolling window: slot 0 = z-1, slot 1 = z (current), slot 2 = z+1 + std::array, 3> maskSlices; + for(auto& ms : maskSlices) + { + ms.resize(sliceSize); + } + // maskCopy uses same rolling window structure for output + std::array, 3> maskCopySlices; + for(auto& ms : maskCopySlices) + { + ms.resize(sliceSize); + } + + // Temporary bool buffer for bulk I/O (std::vector is bit-packed, so use unique_ptr) + auto boolBuf = std::make_unique(sliceSize); + auto& maskStore = mask.getDataStoreRef(); + + auto readMaskSlice = [&](int64 z, usize slot) { + const usize zOffset = static_cast(z) * sliceSize; + maskStore.copyIntoBuffer(zOffset, nonstd::span(boolBuf.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + maskSlices[slot][i] = boolBuf[i] ? 1 : 0; + maskCopySlices[slot][i] = maskSlices[slot][i]; + } + }; + + // Face neighbor ordering: 0=-Z, 1=-Y, 2=-X, 3=+X, 4=+Y, 5=+Z + constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + + for(int32 iteration = 0; iteration < m_InputValues->NumIterations; iteration++) { m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Iteration {}", iteration)); - for(size_t j = 0; j < totalPoints; j++) + // Initialize rolling window: load z=0 into slot 1, z=1 into slot 2 + readMaskSlice(0, 1); + if(dims[2] > 1) { - maskCopy[j] = mask[j]; + readMaskSlice(1, 2); } + for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - const int64 zStride = dims[0] * dims[1] * zIdx; + // Advance rolling window for z > 0 + if(zIdx > 0) + { + std::swap(maskSlices[0], maskSlices[1]); + std::swap(maskSlices[1], maskSlices[2]); + std::swap(maskCopySlices[0], maskCopySlices[1]); + std::swap(maskCopySlices[1], maskCopySlices[2]); + if(zIdx + 1 < dims[2]) + { + readMaskSlice(zIdx + 1, 2); + } + } + for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { - const int64 yStride = dims[0] * yIdx; for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { - const int64 voxelIndex = zStride + yStride + xIdx; + const usize inSlice = static_cast(yIdx * dims[0] + xIdx); - if(!mask[voxelIndex]) + if(maskSlices[1][inSlice] == 0) { - // Loop over the 6 face neighbors of the voxel std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + if(!m_InputValues->XDirOn) + { + isValidFaceNeighbor[k_NegativeXNeighbor] = false; + isValidFaceNeighbor[k_PositiveXNeighbor] = false; + } + if(!m_InputValues->YDirOn) + { + isValidFaceNeighbor[k_NegativeYNeighbor] = false; + isValidFaceNeighbor[k_PositiveYNeighbor] = false; + } + if(!m_InputValues->ZDirOn) + { + isValidFaceNeighbor[k_NegativeZNeighbor] = false; + isValidFaceNeighbor[k_PositiveZNeighbor] = false; + } + + const std::array neighborInSlice = { + inSlice, // -Z: same xy position in prev slice + static_cast((yIdx - 1) * dims[0] + xIdx), // -Y + static_cast(yIdx * dims[0] + (xIdx - 1)), // -X + static_cast(yIdx * dims[0] + (xIdx + 1)), // +X + static_cast((yIdx + 1) * dims[0] + xIdx), // +Y + inSlice // +Z: same xy position in next slice + }; + for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) @@ -76,24 +146,48 @@ Result<> ErodeDilateMask::operator()() continue; } - const int64 neighpoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; - - if(m_InputValues->Operation == detail::k_DilateIndex && mask[neighpoint]) + if(m_InputValues->Operation == detail::k_DilateIndex && maskSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] != 0) { - maskCopy[voxelIndex] = true; + maskCopySlices[1][inSlice] = 1; } - if(m_InputValues->Operation == detail::k_ErodeIndex && mask[neighpoint]) + if(m_InputValues->Operation == detail::k_ErodeIndex && maskSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] != 0) { - maskCopy[neighpoint] = false; + maskCopySlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] = 0; } } } } } + + // Write back the completed z-1 slice using bulk I/O + if(zIdx > 0) + { + const usize prevZOffset = static_cast(zIdx - 1) * sliceSize; + for(usize i = 0; i < sliceSize; i++) + { + boolBuf[i] = (maskCopySlices[0][i] != 0); + } + maskStore.copyFromBuffer(prevZOffset, nonstd::span(boolBuf.get(), sliceSize)); + } + } + + // Write back the last slice(s) using bulk I/O + if(dims[2] == 1) + { + for(usize i = 0; i < sliceSize; i++) + { + boolBuf[i] = (maskCopySlices[1][i] != 0); + } + maskStore.copyFromBuffer(0, nonstd::span(boolBuf.get(), sliceSize)); } - for(size_t j = 0; j < totalPoints; j++) + else { - mask[j] = maskCopy[j]; + const usize lastZOffset = static_cast(dims[2] - 1) * sliceSize; + for(usize i = 0; i < sliceSize; i++) + { + boolBuf[i] = (maskCopySlices[1][i] != 0); + } + maskStore.copyFromBuffer(lastZOffset, nonstd::span(boolBuf.get(), sliceSize)); } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp index c0dc2a5273..1b7ac7bc6e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp @@ -22,6 +22,10 @@ static inline constexpr ChoicesParameter::ValueType k_DilateIndex = 0ULL; static inline constexpr ChoicesParameter::ValueType k_ErodeIndex = 1ULL; } // namespace detail +/** + * @struct ErodeDilateMaskInputValues + * @brief Holds all user-supplied parameters for the ErodeDilateMask algorithm. + */ struct SIMPLNXCORE_EXPORT ErodeDilateMaskInputValues { ChoicesParameter::ValueType Operation; @@ -34,12 +38,24 @@ struct SIMPLNXCORE_EXPORT ErodeDilateMaskInputValues }; /** - * @class + * @class ErodeDilateMask + * @brief Erodes or dilates a boolean mask array using face-neighbor connectivity. */ class SIMPLNXCORE_EXPORT ErodeDilateMask { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays + * @param mesgHandler Handler for sending progress messages to the UI + * @param shouldCancel Atomic flag checked between iterations to support cancellation + * @param inputValues User-supplied parameters controlling the algorithm behavior + */ ErodeDilateMask(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ErodeDilateMaskInputValues* inputValues); + + /** + * @brief Default destructor. + */ ~ErodeDilateMask() noexcept; ErodeDilateMask(const ErodeDilateMask&) = delete; @@ -47,9 +63,13 @@ class SIMPLNXCORE_EXPORT ErodeDilateMask ErodeDilateMask& operator=(const ErodeDilateMask&) = delete; ErodeDilateMask& operator=(ErodeDilateMask&&) noexcept = delete; + /** + * @brief Executes the erode/dilate mask algorithm. + * @return Result<> indicating success or any errors encountered during execution + */ Result<> operator()(); - const std::atomic_bool& getCancel(); + const std::atomic_bool& getCancel() const; private: DataStructure& m_DataStructure; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp new file mode 100644 index 0000000000..e9f4521ff0 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp @@ -0,0 +1,29 @@ +#include "FillBadData.hpp" + +#include "FillBadDataBFS.hpp" +#include "FillBadDataCCL.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +FillBadData::FillBadData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +FillBadData::~FillBadData() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> FillBadData::operator()() +{ + auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath); + + return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp new file mode 100644 index 0000000000..db21d91419 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp @@ -0,0 +1,63 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +#include + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT FillBadDataInputValues +{ + int32 minAllowedDefectSizeValue; + bool storeAsNewPhase; + DataPath featureIdsArrayPath; + DataPath cellPhasesArrayPath; + std::vector ignoredDataArrayPaths; + DataPath inputImageGeometry; +}; + +/** + * @class FillBadData + * @brief Dispatcher that selects between BFS (in-core) and CCL (out-of-core) algorithms. + * + * @see FillBadDataBFS for the in-core-optimized implementation. + * @see FillBadDataCCL for the out-of-core-optimized implementation. + * @see AlgorithmDispatch.hpp for the dispatch mechanism. + */ +class SIMPLNXCORE_EXPORT FillBadData +{ +public: + /** + * @brief Constructs the dispatcher with the required context for algorithm selection. + * @param dataStructure The data structure containing the arrays to process. + * @param mesgHandler Handler for progress and informational messages. + * @param shouldCancel Cancellation flag checked during execution. + * @param inputValues Filter parameter values controlling fill behavior. + */ + FillBadData(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues); + ~FillBadData() noexcept; + + FillBadData(const FillBadData&) = delete; + FillBadData(FillBadData&&) noexcept = delete; + FillBadData& operator=(const FillBadData&) = delete; + FillBadData& operator=(FillBadData&&) noexcept = delete; + + /** + * @brief Dispatches to either BFS or CCL algorithm based on data residency. + * @return Result indicating success or an error with a descriptive message. + */ + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const FillBadDataInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp index 6eb72ef971..63051e4d9e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp @@ -1,80 +1,52 @@ #include "FillBadDataBFS.hpp" +#include "FillBadData.hpp" + #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/DataGroupUtilities.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/MessageHelper.hpp" -#include "simplnx/Utilities/NeighborUtilities.hpp" -#include -#include +#include using namespace nx::core; -// ============================================================================= -// FillBadDataBFS Algorithm Overview -// ============================================================================= -// -// This file implements an optimized algorithm for filling bad data (voxels with -// FeatureId == 0) in image geometries. The algorithm handles out-of-core datasets -// efficiently by processing data in chunks and uses a four-phase approach: -// -// Phase 1: Chunk-Sequential Connected Component Labeling (CCL) -// - Process chunks sequentially, assigning provisional labels to bad data regions -// - Use Union-Find to track equivalences between labels across chunk boundaries -// - Track size of each connected component -// -// Phase 2: Global Resolution -// - Flatten Union-Find structure to resolve all equivalences -// - Accumulate region sizes to root labels -// -// Phase 3: Region Classification and Relabeling -// - Classify regions as "small" (below threshold) or "large" (above threshold) -// - Small regions: mark with -1 for filling in Phase 4 -// - Large regions: keep as 0 or assign to new phase (if requested) -// -// Phase 4: Iterative Morphological Fill -// - Iteratively fill -1 voxels by assigning them to the most common neighbor -// - Update all cell data arrays to match the filled voxels -// -// ============================================================================= - namespace { // ----------------------------------------------------------------------------- -// Helper function: Update data array tuples based on neighbor assignments +// FillBadDataUpdateTuples // ----------------------------------------------------------------------------- -// Copies data from neighbor voxels to fill bad data voxels (-1 values) -// This is used to propagate cell data attributes during the filling process +// Copies cell data array values from a good neighbor voxel to each bad data +// voxel. The `neighbors` vector maps each voxel index to the index of its best +// source neighbor (determined by majority vote in the iterative fill loop). +// +// Only voxels satisfying ALL of the following conditions are updated: +// - featureId < 0 (marked as small bad-data region needing fill) +// - neighbor != -1 (a valid source neighbor was found) +// - neighbor != tupleIndex (not self-referencing; default sentinel) +// - featureIds[neighbor] > 0 (the source is a real feature, not bad data) // -// @param featureIds The feature IDs array indicating which voxels are bad data -// @param outputDataStore The data array to update -// @param neighbors The neighbor assignments (index of the neighbor to copy from) +// All components of the tuple are copied (e.g., 3-component RGB, 6-component +// tensor, etc.), preserving multi-component array semantics. +// ----------------------------------------------------------------------------- template -void FillBadDataBFSUpdateTuples(const Int32AbstractDataStore& featureIds, AbstractDataStore& outputDataStore, const std::vector& neighbors) +void FillBadDataUpdateTuples(const Int32AbstractDataStore& featureIds, AbstractDataStore& outputDataStore, const std::vector& neighbors) { usize start = 0; usize stop = outputDataStore.getNumberOfTuples(); const usize numComponents = outputDataStore.getNumberOfComponents(); - - // Loop through all tuples in the data array for(usize tupleIndex = start; tupleIndex < stop; tupleIndex++) { const int32 featureName = featureIds[tupleIndex]; const int32 neighbor = neighbors[tupleIndex]; - - // Skip if no neighbor assignment if(neighbor == tupleIndex) { continue; } - // Copy data from the valid neighbor to bad data voxel - // Only copy if the current voxel is bad data (-1) and the neighbor is valid (>0) if(featureName < 0 && neighbor != -1 && featureIds[static_cast(neighbor)] > 0) { - // Copy all components from neighbor tuple to current tuple for(usize i = 0; i < numComponents; i++) { auto value = outputDataStore[neighbor * numComponents + i]; @@ -84,182 +56,19 @@ void FillBadDataBFSUpdateTuples(const Int32AbstractDataStore& featureIds, Abstra } } -// ----------------------------------------------------------------------------- -// Functor for type-dispatched tuple updates -// ----------------------------------------------------------------------------- -// Allows the FillBadDataBFSUpdateTuples function to be called with runtime type dispatch -struct FillBadDataBFSUpdateTuplesFunctor +struct FillBadDataUpdateTuplesFunctor { template void operator()(const Int32AbstractDataStore& featureIds, IDataArray* outputIDataArray, const std::vector& neighbors) { auto& outputStore = outputIDataArray->template getIDataStoreRefAs>(); - FillBadDataBFSUpdateTuples(featureIds, outputStore, neighbors); + FillBadDataUpdateTuples(featureIds, outputStore, neighbors); } }; } // namespace -// ============================================================================= -// ChunkAwareUnionFind Implementation -// ============================================================================= -// -// A Union-Find (Disjoint Set) data structure optimized for tracking connected -// component equivalences during chunk-sequential processing. Uses union-by-rank -// for efficient merging and defers path compression to a single flatten() pass -// to avoid redundant updates during construction. -// -// Key features: -// - Lazily creates entries as labels are encountered -// - Tracks rank for balanced union operations -// - Accumulates sizes at each label (not root) during construction -// - Single-pass path compression and size accumulation in flatten() -// ============================================================================= - -// ----------------------------------------------------------------------------- -// Find the root representative of a label's equivalence class -// ----------------------------------------------------------------------------- -// This performs a simple root lookup without path compression. Path compression -// is deferred to the flatten() method to avoid wasting cycles updating paths -// that will be modified again during later merges. -// -// @param x The label to find the root for -// @return The root label of the equivalence class -int64 ChunkAwareUnionFind::find(int64 x) -{ - // Create a parent entry if it doesn't exist (lazy initialization) - if(!m_Parent.contains(x)) - { - m_Parent[x] = x; - m_Rank[x] = 0; - m_Size[x] = 0; - } - - // Find root iteratively without using the path compression algorithm - // Path compression is deferred to flatten() to avoid wasting cycles - // during frequent merges where paths would be updated repeatedly - int64 root = x; - while(m_Parent[root] != root) - { - root = m_Parent[root]; - } - - return root; -} - -// ----------------------------------------------------------------------------- -// Unite two labels into the same equivalence class -// ----------------------------------------------------------------------------- -// Merges the sets containing labels a and b using union-by-rank heuristic. -// This keeps the tree balanced for better performance. -// -// @param a First label -// @param b Second label -void ChunkAwareUnionFind::unite(int64 a, int64 b) -{ - int64 rootA = find(a); - int64 rootB = find(b); - - // Already in the same set - if(rootA == rootB) - { - return; - } - - // Union by rank: attach the smaller tree object under the root of the larger tree - // This keeps the tree height logarithmic for better find() performance - if(m_Rank[rootA] < m_Rank[rootB]) - { - m_Parent[rootA] = rootB; - } - else if(m_Rank[rootA] > m_Rank[rootB]) - { - m_Parent[rootB] = rootA; - } - else - { - // Equal rank: arbitrarily choose rootA as the parent and increment its rank - m_Parent[rootB] = rootA; - m_Rank[rootA]++; - } -} - -// ----------------------------------------------------------------------------- -// Add voxel count to a label's size -// ----------------------------------------------------------------------------- -// During construction, sizes are accumulated at each label (not root). -// This allows concurrent size updates without needing to find roots. -// All sizes will be accumulated to roots during flatten(). -// -// @param label The label to add size to -// @param count Number of voxels to add -void ChunkAwareUnionFind::addSize(int64 label, uint64 count) -{ - // Add size to the label itself, not the root - // Sizes will be accumulated to roots during flatten() - m_Size[label] += count; -} - -// ----------------------------------------------------------------------------- -// Get the total size of a label's equivalence class -// ----------------------------------------------------------------------------- -// Returns the accumulated size for a label's root. Should only be called -// after flatten() has been executed to get accurate totals. -// -// @param label The label to query -// @return Total number of voxels in the equivalence class -uint64 ChunkAwareUnionFind::getSize(int64 label) -{ - int64 root = find(label); - auto it = m_Size.find(root); - if(it == m_Size.end()) - { - return 0; - } - return it->second; -} - // ----------------------------------------------------------------------------- -// Flatten the Union-Find structure with path compression -// ----------------------------------------------------------------------------- -// Performs a single-pass path compression and size accumulation after all -// merges are complete. This is more efficient than doing path compression -// during every find() operation when there are frequent merges. -// -// After flatten(): -// - Every label points directly to its root (fully compressed paths) -// - All sizes are accumulated at root labels -// - Subsequent find() and getSize() operations are O(1) -void ChunkAwareUnionFind::flatten() -{ - // First pass: flatten all parents with path compression - // Make every label point directly to its root for O(1) lookups - // This is done in a single pass after all merges to avoid wasting - // cycles updating paths repeatedly during construction - std::unordered_map finalRoots; - for(auto& [label, parent] : m_Parent) - { - int64 root = find(label); - finalRoots[label] = root; - } - - // Second pass: accumulate sizes to roots - // Sum up all the sizes from individual labels to their root representatives - std::unordered_map rootSizes; - for(const auto& [label, root] : finalRoots) - { - rootSizes[root] += m_Size[label]; - } - - // Replace maps with flattened versions for O(1) access - m_Parent = finalRoots; - m_Size = rootSizes; -} - -// ============================================================================= -// FillBadDataBFS Implementation -// ============================================================================= - -FillBadDataBFS::FillBadDataBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, FillBadDataInputValues* inputValues) +FillBadDataBFS::FillBadDataBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -271,275 +80,193 @@ FillBadDataBFS::FillBadDataBFS(DataStructure& dataStructure, const IFilter::Mess FillBadDataBFS::~FillBadDataBFS() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& FillBadDataBFS::getCancel() const -{ - return m_ShouldCancel; -} - -// ============================================================================= -// PHASE 1: Chunk-Sequential Connected Component Labeling (CCL) -// ============================================================================= +// FillBadDataBFS::operator() +// ----------------------------------------------------------------------------- +// BFS-based flood-fill algorithm for replacing bad data voxels with values +// from neighboring good features. The algorithm has three main steps: // -// Performs connected component labeling on bad data voxels (FeatureId == 0) -// using a chunk-sequential scanline algorithm. This approach is optimized for -// out-of-core datasets where data is stored in chunks on the disk. +// Step 1: Find the maximum feature ID (and optionally maximum phase). // -// Algorithm: -// 1. Process chunks sequentially, loading one chunk at a time -// 2. For each bad data voxel, check already-processed neighbors (-X, -Y, -Z) -// 3. If neighbors exist, reuse their label; otherwise assign new label -// 4. Track label equivalences in Union-Find structure -// 5. Track size of each connected component +// Step 2: BFS flood-fill to discover connected regions of bad data +// (featureId == 0). Each region is classified by size: +// - Large regions (>= minAllowedDefectSize): kept as voids (featureId +// stays 0, optionally assigned a new phase). +// - Small regions (< threshold): marked with featureId = -1 for filling. // -// The scanline order ensures we only need to check 3 neighbors (previous in -// X, Y, and Z directions) instead of all 6 face neighbors, because later -// neighbors haven't been processed yet. +// Step 3: Iterative morphological dilation. Each iteration scans all -1 +// voxels, finds the neighboring good feature with the most face-adjacent +// votes (majority vote), and records the best neighbor. Then copies all +// cell data components from that neighbor to the -1 voxel. Repeats until +// no -1 voxels remain. FeatureIds are updated LAST to avoid changing the +// vote source mid-iteration. // -// @param featureIdsStore The feature IDs data store (maybe out-of-core) -// @param unionFind Union-Find structure for tracking label equivalences -// @param provisionalLabels Map from voxel index to assigned provisional label -// @param dims Image dimensions [X, Y, Z] -// ============================================================================= -Result<> FillBadDataBFS::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims) +// NOTE: This algorithm uses O(N) memory (neighbors + alreadyChecked + +// featureNumber vectors), making it unsuitable for very large OOC datasets. +// Use FillBadDataCCL for out-of-core compatible processing. +// ----------------------------------------------------------------------------- +Result<> FillBadDataBFS::operator()() { - int64 nextLabel = -1; - const usize slabSize = static_cast(dims[0]) * static_cast(dims[1]); + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath)->getDataStoreRef(); + const usize totalPoints = featureIdsStore.getNumberOfTuples(); - // Two slab buffers: current Z-slab and previous Z-slab for -Z neighbor checks - std::vector curSlab(slabSize); - std::vector prevSlab(slabSize); + // O(N) allocations: one int32 per voxel for neighbor mapping, one bit per + // voxel for BFS visited tracking + std::vector neighbors(totalPoints, -1); + std::vector alreadyChecked(totalPoints, false); - for(int64 z = 0; z < dims[2]; z++) - { - const usize slabStart = static_cast(z) * slabSize; - auto readResult = featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(curSlab.data(), slabSize)); - if(readResult.invalid()) - { - return MergeResults(readResult, - MakeErrorResult(-71500, fmt::format("FillBadData phase 1 (connected component labeling): failed to read Z-slab {} (start index {}, size {}) from feature IDs store.", z, - slabStart, slabSize))); - } + const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); + const SizeVec3 udims = selectedImageGeom.getDimensions(); - for(int64 y = 0; y < dims[1]; y++) - { - for(int64 x = 0; x < dims[0]; x++) - { - const usize localIdx = static_cast(y) * static_cast(dims[0]) + static_cast(x); - if(curSlab[localIdx] != 0) - { - continue; - } + Int32Array* cellPhasesPtr = nullptr; - const usize globalIdx = slabStart + localIdx; - std::vector neighborLabels; + if(m_InputValues->storeAsNewPhase) + { + cellPhasesPtr = m_DataStructure.getDataAs(m_InputValues->cellPhasesArrayPath); + } - // Check -X neighbor (same slab) - if(x > 0 && curSlab[localIdx - 1] == 0) - { - const usize nIdx = globalIdx - 1; - if(provisionalLabels.contains(nIdx)) - { - neighborLabels.push_back(provisionalLabels[nIdx]); - } - } - // Check -Y neighbor (same slab) - if(y > 0 && curSlab[localIdx - dims[0]] == 0) - { - const usize nIdx = globalIdx - dims[0]; - if(provisionalLabels.contains(nIdx)) - { - neighborLabels.push_back(provisionalLabels[nIdx]); - } - } - // Check -Z neighbor (previous slab) - if(z > 0 && prevSlab[localIdx] == 0) - { - const usize nIdx = globalIdx - slabSize; - if(provisionalLabels.contains(nIdx)) - { - neighborLabels.push_back(provisionalLabels[nIdx]); - } - } + std::array dims = { + static_cast(udims[0]), + static_cast(udims[1]), + static_cast(udims[2]), + }; - int64 assignedLabel = 0; - if(neighborLabels.empty()) - { - assignedLabel = nextLabel--; - unionFind.find(assignedLabel); - } - else - { - assignedLabel = neighborLabels[0]; - for(usize i = 1; i < neighborLabels.size(); i++) - { - if(neighborLabels[i] != assignedLabel) - { - unionFind.unite(assignedLabel, neighborLabels[i]); - } - } - } + usize count = 1; + usize numFeatures = 0; + usize maxPhase = 0; - provisionalLabels[globalIdx] = assignedLabel; - unionFind.addSize(assignedLabel, 1); - } + // --- Step 1: Find the maximum feature ID across all voxels ---------------- + // This value is used to size the featureNumber vote counter in Step 3. + for(usize i = 0; i < totalPoints; i++) + { + int32 featureName = featureIdsStore[i]; + if(featureName > numFeatures) + { + numFeatures = featureName; } - - std::swap(prevSlab, curSlab); } - return {}; -} -// ============================================================================= -// PHASE 2: Global Resolution of Equivalences -// ============================================================================= -// -// Resolves all label equivalences from Phase 1 and accumulates region sizes. -// After this phase: -// - All labels point directly to their root representatives -// - All sizes are accumulated at root labels -// - Region sizes can be queried in O(1) time -// -// @param unionFind Union-Find structure containing label equivalences -// @param smallRegions Unused in current implementation (kept for interface compatibility) -// ============================================================================= -void FillBadDataBFS::phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std::unordered_set& smallRegions) -{ - // Flatten the union-find structure to: - // 1. Compress all paths (make every label point directly to root) - // 2. Accumulate all sizes to root labels - unionFind.flatten(); -} - -// ============================================================================= -// PHASE 3: Region Classification and Relabeling -// ============================================================================= -// -// Classifies bad data regions as "small" or "large" based on size threshold: -// - Small regions (< minAllowedDefectSize): marked with -1 for filling in Phase 4 -// - Large regions (>= minAllowedDefectSize): kept as 0 (or assigned new phase) -// -// This phase processes chunks to relabel voxels based on their region classification. -// Large regions may optionally be assigned to a new phase (if storeAsNewPhase is true). -// -// @param featureIdsStore The feature IDs data store -// @param cellPhasesPtr Cell phases array (maybe null) -// @param provisionalLabels Map from voxel index to provisional label (from Phase 1) -// @param smallRegions Unused in current implementation (kept for interface compatibility) -// @param unionFind Union-Find structure with resolved equivalences (from Phase 2) -// @param maxPhase Maximum existing phase value (for new phase assignment) -// ============================================================================= -Result<> FillBadDataBFS::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, - const std::unordered_set& /*smallRegions*/, ChunkAwareUnionFind& unionFind, usize maxPhase) const -{ - // Classify regions by size - std::unordered_map rootSizes; - for(const auto& [index, label] : provisionalLabels) + // Optionally find the maximum phase so large void regions can be assigned + // to (maxPhase + 1), creating a distinct phase for visualization. + if(m_InputValues->storeAsNewPhase) { - int64 root = unionFind.find(label); - if(!rootSizes.contains(root)) + for(usize i = 0; i < totalPoints; i++) { - rootSizes[root] = unionFind.getSize(root); + if((*cellPhasesPtr)[i] > maxPhase) + { + maxPhase = (*cellPhasesPtr)[i]; + } } } - std::unordered_set localSmallRegions; - for(const auto& [root, size] : rootSizes) + // Face-neighbor offsets in flat index space: -Z, -Y, -X, +X, +Y, +Z + std::array neighborPoints = {-dims[0] * dims[1], -dims[0], -1, 1, dims[0], dims[0] * dims[1]}; + std::vector currentVisitedList; + + MessageHelper messageHelper(m_MessageHandler); + + // --- Step 2: BFS flood-fill to classify bad data regions ------------------ + // Mark all non-zero voxels as already checked (they are good features). + // Then BFS from each unchecked voxel with featureId == 0 to discover + // contiguous bad data regions. + for(usize iter = 0; iter < totalPoints; iter++) { - if(static_cast(size) < m_InputValues->minAllowedDefectSizeValue) + alreadyChecked[iter] = false; + if(featureIdsStore[iter] != 0) { - localSmallRegions.insert(root); + alreadyChecked[iter] = true; } } - // Process slab-by-slab, reading and writing back via bulk I/O - const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); - const SizeVec3 udims = selectedImageGeom.getDimensions(); - const usize slabSize = udims[0] * udims[1]; - std::vector slab(slabSize); + messageHelper.sendMessage("Identifying bad data regions via BFS..."); - for(usize z = 0; z < udims[2]; z++) + for(usize i = 0; i < totalPoints; i++) { - const usize slabStart = z * slabSize; - auto readResult = featureIdsStore.copyIntoBuffer(slabStart, nonstd::span(slab.data(), slabSize)); - if(readResult.invalid()) + if(m_ShouldCancel) { - return MergeResults(readResult, MakeErrorResult(-71501, fmt::format("FillBadData phase 3 (region classification): failed to read Z-slab {} (start index {}, size {}) from feature IDs store.", z, - slabStart, slabSize))); + return {}; } - bool slabModified = false; - for(usize localIdx = 0; localIdx < slabSize; localIdx++) + if(!alreadyChecked[i] && featureIdsStore[i] == 0) { - const usize globalIdx = slabStart + localIdx; - auto labelIter = provisionalLabels.find(globalIdx); - if(labelIter == provisionalLabels.end()) - { - continue; - } - - int64 root = unionFind.find(labelIter->second); - if(localSmallRegions.contains(root)) + // Start a new BFS from this seed voxel to discover all connected + // bad-data voxels in this region + currentVisitedList.push_back(static_cast(i)); + count = 0; + while(count < currentVisitedList.size()) { - slab[localIdx] = -1; + int64 index = currentVisitedList[count]; + int64 column = index % dims[0]; + int64 row = (index / dims[0]) % dims[1]; + int64 plane = index / (dims[0] * dims[1]); + // Check all 6 face-adjacent neighbors, with boundary guard checks + for(int32 j = 0; j < 6; j++) + { + int64 neighbor = index + neighborPoints[j]; + if(j == 0 && plane == 0) + { + continue; + } + if(j == 5 && plane == (dims[2] - 1)) + { + continue; + } + if(j == 1 && row == 0) + { + continue; + } + if(j == 4 && row == (dims[1] - 1)) + { + continue; + } + if(j == 2 && column == 0) + { + continue; + } + if(j == 3 && column == (dims[0] - 1)) + { + continue; + } + if(featureIdsStore[neighbor] == 0 && !alreadyChecked[neighbor]) + { + currentVisitedList.push_back(neighbor); + alreadyChecked[neighbor] = true; + } + } + count++; } - else + // Classify this region by size: + // Large regions (>= threshold): keep as voids (featureId = 0), + // optionally assign to a new phase for visualization. + if((int32)currentVisitedList.size() >= m_InputValues->minAllowedDefectSizeValue) { - slab[localIdx] = 0; - if(m_InputValues->storeAsNewPhase && cellPhasesPtr != nullptr) + for(const auto& currentIndex : currentVisitedList) { - (*cellPhasesPtr)[globalIdx] = static_cast(maxPhase) + 1; + featureIdsStore[currentIndex] = 0; + if(m_InputValues->storeAsNewPhase) + { + (*cellPhasesPtr)[currentIndex] = static_cast(maxPhase) + 1; + } } } - slabModified = true; - } - - if(slabModified) - { - auto writeResult = featureIdsStore.copyFromBuffer(slabStart, nonstd::span(slab.data(), slabSize)); - if(writeResult.invalid()) + // Small regions (< threshold): mark with -1 to indicate they should + // be filled in Step 3 by copying data from neighboring good features. + if((int32)currentVisitedList.size() < m_InputValues->minAllowedDefectSizeValue) { - return MergeResults(writeResult, - MakeErrorResult(-71502, fmt::format("FillBadData phase 3 (region classification): failed to write Z-slab {} (start index {}, size {}) back to feature IDs store.", z, - slabStart, slabSize))); + for(const auto& currentIndex : currentVisitedList) + { + featureIdsStore[currentIndex] = -1; + } } + currentVisitedList.clear(); } } - return {}; -} -// ============================================================================= -// PHASE 4: Iterative Morphological Fill -// ============================================================================= -// -// Fills small bad data regions (marked with -1 in Phase 3) using iterative -// morphological dilation. Each iteration: -// 1. For each -1 voxel, find the most common positive feature among its neighbors -// 2. Assign that voxel to the most common neighbor's feature -// 3. Update all cell data arrays to match the filled voxels -// -// This process repeats until all -1 voxels have been filled. The algorithm -// gradually fills small defects from the edges inward, ensuring smooth boundaries. -// -// @param featureIdsStore The feature IDs data store -// @param dims Image dimensions [X, Y, Z] -// @param numFeatures Number of features in the dataset -// ============================================================================= -void FillBadDataBFS::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, usize numFeatures) const -{ - const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); - const usize totalPoints = featureIdsStore.getNumberOfTuples(); - - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - - // Neighbor assignment array: neighbors[i] = index of the neighbor to copy from - std::vector neighbors(totalPoints, -1); - - // Feature vote counter: tracks how many times each feature appears as the neighbor + // --- Step 3: Iterative morphological dilation ----------------------------- + // Vote counter indexed by feature ID. O(numFeatures) memory. std::vector featureNumber(numFeatures + 1, 0); - // Get a list of all cell arrays that need to be updated during filling - // Exclude arrays specified in ignoredDataArrayPaths + // Collect all cell data arrays that need updating when a voxel is filled + // (excludes user-specified ignored arrays) std::optional> allChildArrays = GetAllChildDataPaths(m_DataStructure, selectedImageGeom.getCellDataPath(), DataObject::Type::DataArray, m_InputValues->ignoredDataArrayPaths); std::vector voxelArrayNames; if(allChildArrays.has_value()) @@ -547,80 +274,105 @@ void FillBadDataBFS::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsSt voxelArrayNames = allChildArrays.value(); } - // Create a message helper for throttled progress updates (1 update per second) - MessageHelper messageHelper(m_MessageHandler, std::chrono::milliseconds(1000)); - auto throttledMessenger = messageHelper.createThrottledMessenger(std::chrono::milliseconds(1000)); - - usize count = 1; // Number of voxels with -1 value that remain - usize iteration = 0; // Current iteration number - - // Iteratively fill until no voxels with -1 value remain + // Iterate until no -1 voxels remain. Each iteration grows the good-data + // boundary inward by one voxel layer (morphological dilation). + int32 iteration = 0; while(count != 0) { - iteration++; - count = 0; // Reset count of voxels with a -1 value for this iteration - - // Pass 1: Determine neighbor assignments for all -1 voxels - // For each -1 voxel, find the most common positive feature among neighbors - for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) + if(m_ShouldCancel) { - int32 featureName = featureIdsStore[voxelIndex]; + return {}; + } - // Only process voxels marked for filling (-1) + iteration++; + count = 0; + for(usize i = 0; i < totalPoints; i++) + { + int32 featureName = featureIdsStore[i]; if(featureName < 0) { - count++; // Count this voxel as needing filling - int32 most = 0; // Highest vote count seen so far - - // Compute 3D position from the linear index - int64 xIdx = voxelIndex % dims[0]; - int64 yIdx = (voxelIndex / dims[0]) % dims[1]; - int64 zIdx = voxelIndex / (dims[0] * dims[1]); - - // Vote for the most common positive neighbor feature - // Loop over the 6 face neighbors of the voxel - std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(const auto& faceIndex : faceNeighborInternalIdx) + count++; + int32 most = 0; + int64 xIndex = static_cast(i % dims[0]); + int64 yIndex = static_cast((i / dims[0]) % dims[1]); + int64 zIndex = static_cast(i / (dims[0] * dims[1])); + + // First neighbor loop: tally votes from face-adjacent good features. + // Each good neighbor increments featureNumber[its featureId]. The + // feature with the highest vote count wins (majority vote), and + // neighbors[i] records the winning neighbor's voxel index. + for(int32 j = 0; j < 6; j++) { - // Skip neighbors outside image bounds - if(!isValidFaceNeighbor[faceIndex]) + auto neighborPoint = static_cast(i + neighborPoints[j]); + if(j == 0 && zIndex == 0) + { + continue; + } + if(j == 5 && zIndex == (dims[2] - 1)) + { + continue; + } + if(j == 1 && yIndex == 0) + { + continue; + } + if(j == 4 && yIndex == (dims[1] - 1)) + { + continue; + } + if(j == 2 && xIndex == 0) + { + continue; + } + if(j == 3 && xIndex == (dims[0] - 1)) { continue; } - auto neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; int32 feature = featureIdsStore[neighborPoint]; - - // Only vote for positive features (valid data) if(feature > 0) { - // Increment vote count for this feature featureNumber[feature]++; int32 current = featureNumber[feature]; - - // Track the feature with the most votes if(current > most) { most = current; - neighbors[voxelIndex] = static_cast(neighborPoint); // Store neighbor to copy from + neighbors[i] = static_cast(neighborPoint); } } } - - // Reset vote counters for next voxel - // Only reset features that were actually counted to save time - // Loop over the 6 face neighbors of the voxel - isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); - for(const auto& faceIndex : faceNeighborInternalIdx) + // Second neighbor loop: reset the vote counters for only the features + // that were incremented above. This avoids zeroing the entire + // featureNumber vector (which would be O(numFeatures) per voxel). + for(int32 j = 0; j < 6; j++) { - if(!isValidFaceNeighbor[faceIndex]) + int64 neighborPoint = static_cast(i) + neighborPoints[j]; + if(j == 0 && zIndex == 0) + { + continue; + } + if(j == 5 && zIndex == (dims[2] - 1)) + { + continue; + } + if(j == 1 && yIndex == 0) + { + continue; + } + if(j == 4 && yIndex == (dims[1] - 1)) + { + continue; + } + if(j == 2 && xIndex == 0) + { + continue; + } + if(j == 3 && xIndex == (dims[0] - 1)) { continue; } - int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; int32 feature = featureIdsStore[neighborPoint]; - if(feature > 0) { featureNumber[feature] = 0; @@ -629,119 +381,25 @@ void FillBadDataBFS::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsSt } } - // Pass 2: Update all cell data arrays based on neighbor assignments - // This propagates all cell data attributes (not just feature IDs) to filled voxels + // Apply fills: update all non-featureIds cell arrays first by copying + // all components from the winning neighbor to the bad voxel. for(const auto& cellArrayPath : voxelArrayNames) { - // Skip the feature IDs array (will be updated separately below) if(cellArrayPath == m_InputValues->featureIdsArrayPath) { continue; } - auto* oldCellArray = m_DataStructure.getDataAs(cellArrayPath); - // Use the type-dispatched update function to handle all data types - ExecuteDataFunction(FillBadDataBFSUpdateTuplesFunctor{}, oldCellArray->getDataType(), featureIdsStore, oldCellArray, neighbors); + ExecuteDataFunction(FillBadDataUpdateTuplesFunctor{}, oldCellArray->getDataType(), featureIdsStore, oldCellArray, neighbors); } - // Update FeatureIds array last to finalize the iteration - FillBadDataBFSUpdateTuples(featureIdsStore, featureIdsStore, neighbors); - - // Send throttled progress update (max 1 per second) - throttledMessenger.sendThrottledMessage([iteration, count]() { return fmt::format(" Iteration {}: {} voxels remaining to fill", iteration, count); }); + // Update FeatureIds LAST: the FillBadDataUpdateTuples calls above rely + // on featureIds to check that the source neighbor is still a valid good + // feature (featureId > 0). If featureIds were updated first, a freshly + // filled voxel could become a vote source before its other arrays were + // copied, leading to inconsistent data. + FillBadDataUpdateTuples(featureIdsStore, featureIdsStore, neighbors); } - - // Send final completion summary - m_MessageHandler({IFilter::Message::Type::Info, fmt::format(" Completed in {} iteration{}", iteration, iteration == 1 ? "" : "s")}); -} - -// ============================================================================= -// Main Algorithm Entry Point -// ============================================================================= -// -// Executes the four-phase bad data filling algorithm: -// 1. Chunk-Sequential CCL: Label connected components of bad data -// 2. Global Resolution: Resolve equivalences and accumulate sizes -// 3. Region Classification: Classify regions as small or large -// 4. Iterative Fill: Fill small regions using morphological dilation -// -// @return Result indicating success or failure -// ============================================================================= -Result<> FillBadDataBFS::operator()() const -{ - auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath)->getDataStoreRef(); - const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); - const SizeVec3 udims = selectedImageGeom.getDimensions(); - - std::array dims = { - static_cast(udims[0]), - static_cast(udims[1]), - static_cast(udims[2]), - }; - - const usize totalPoints = featureIdsStore.getNumberOfTuples(); - - Int32Array* cellPhasesPtr = nullptr; - usize maxPhase = 0; - if(m_InputValues->storeAsNewPhase) - { - cellPhasesPtr = m_DataStructure.getDataAs(m_InputValues->cellPhasesArrayPath); - for(usize i = 0; i < totalPoints; i++) - { - if((*cellPhasesPtr)[i] > maxPhase) - { - maxPhase = (*cellPhasesPtr)[i]; - } - } - } - - usize numFeatures = 0; - { - const usize bufSize = 65536; - std::vector buf(bufSize); - for(usize offset = 0; offset < totalPoints; offset += bufSize) - { - const usize count = std::min(bufSize, totalPoints - offset); - auto readResult = featureIdsStore.copyIntoBuffer(offset, nonstd::span(buf.data(), count)); - if(readResult.invalid()) - { - return MergeResults(readResult, - MakeErrorResult(-71503, fmt::format("FillBadData: failed to scan feature IDs store for maximum feature id (chunk [{}, {}) of {}).", offset, offset + count, totalPoints))); - } - for(usize i = 0; i < count; i++) - { - if(buf[i] > static_cast(numFeatures)) - { - numFeatures = buf[i]; - } - } - } - } - - ChunkAwareUnionFind unionFind; - std::unordered_map provisionalLabels; - std::unordered_set smallRegions; - - m_MessageHandler({IFilter::Message::Type::Info, "Phase 1/4: Labeling connected components..."}); - auto phaseOneResult = phaseOneCCL(featureIdsStore, unionFind, provisionalLabels, dims); - if(phaseOneResult.invalid()) - { - return phaseOneResult; - } - - m_MessageHandler({IFilter::Message::Type::Info, "Phase 2/4: Resolving region equivalences..."}); - phaseTwoGlobalResolution(unionFind, smallRegions); - - m_MessageHandler({IFilter::Message::Type::Info, "Phase 3/4: Classifying region sizes..."}); - auto phaseThreeResult = phaseThreeRelabeling(featureIdsStore, cellPhasesPtr, provisionalLabels, smallRegions, unionFind, maxPhase); - if(phaseThreeResult.invalid()) - { - return phaseThreeResult; - } - - m_MessageHandler({IFilter::Message::Type::Info, "Phase 4/4: Filling small defects..."}); - phaseFourIterativeFill(featureIdsStore, dims, numFeatures); - return {}; } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp index e7679dd366..6291ce0e3c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp @@ -1,4 +1,3 @@ - #pragma once #include "SimplnxCore/SimplnxCore_export.hpp" @@ -7,89 +6,34 @@ #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include -#include -#include - namespace nx::core { -// Forward declarations -template -class DataArray; -using Int32Array = DataArray; - -template -class AbstractDataStore; -using Int32AbstractDataStore = AbstractDataStore; - -/** - * @class ChunkAwareUnionFind - * @brief Union-Find data structure for tracking connected component equivalences across chunks - */ -class SIMPLNXCORE_EXPORT ChunkAwareUnionFind -{ -public: - ChunkAwareUnionFind() = default; - ~ChunkAwareUnionFind() = default; - - /** - * @brief Find the root label with path compression - * @param x Label to find - * @return Root label - */ - int64 find(int64 x); - - /** - * @brief Unite two labels into the same equivalence class - * @param a First label - * @param b Second label - */ - void unite(int64 a, int64 b); - - /** - * @brief Add to the size count for a label - * @param label Label to update - * @param count Number of voxels to add - */ - void addSize(int64 label, uint64 count); - - /** - * @brief Get the total size of a label's equivalence class - * @param label Label to query - * @return Total number of voxels in the equivalence class - */ - uint64 getSize(int64 label); - - /** - * @brief Flatten the union-find structure and sum sizes to roots - */ - void flatten(); - -private: - std::unordered_map m_Parent; - std::unordered_map m_Rank; - std::unordered_map m_Size; -}; - -struct SIMPLNXCORE_EXPORT FillBadDataInputValues -{ - int32 minAllowedDefectSizeValue; - bool storeAsNewPhase; - DataPath featureIdsArrayPath; - DataPath cellPhasesArrayPath; - std::vector ignoredDataArrayPaths; - DataPath inputImageGeometry; -}; +struct FillBadDataInputValues; /** * @class FillBadDataBFS - + * @brief BFS flood-fill algorithm for filling bad data regions. + * + * This is the in-core-optimized implementation. It uses BFS (breadth-first search) + * to identify connected components of bad data, then iteratively fills small regions + * by voting among face neighbors. Uses O(N) temporary buffers (neighbors, alreadyChecked) + * which is efficient when data fits in RAM. + * + * @see FillBadDataCCL for the out-of-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT FillBadDataBFS { public: - FillBadDataBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, FillBadDataInputValues* inputValues); + /** + * @brief Constructs the BFS fill algorithm with the required context. + * @param dataStructure The data structure containing the arrays to process. + * @param mesgHandler Handler for progress and informational messages. + * @param shouldCancel Cancellation flag checked during execution. + * @param inputValues Filter parameter values controlling fill behavior. + */ + FillBadDataBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues); ~FillBadDataBFS() noexcept; FillBadDataBFS(const FillBadDataBFS&) = delete; @@ -97,49 +41,13 @@ class SIMPLNXCORE_EXPORT FillBadDataBFS FillBadDataBFS& operator=(const FillBadDataBFS&) = delete; FillBadDataBFS& operator=(FillBadDataBFS&&) noexcept = delete; - Result<> operator()() const; - - const std::atomic_bool& getCancel() const; - -private: - /** - * @brief Phase 1: Chunk-sequential connected component labeling - * @param featureIdsStore Feature IDs data store - * @param unionFind Union-find structure for tracking equivalences - * @param provisionalLabels Map from voxel index to provisional label - * @param dims Image geometry dimensions - * @return Result<> invalid if a bulk read from the feature IDs store fails. - */ - static Result<> phaseOneCCL(Int32AbstractDataStore& featureIdsStore, ChunkAwareUnionFind& unionFind, std::unordered_map& provisionalLabels, const std::array& dims); - /** - * @brief Phase 2: Global resolution of equivalences and region classification - * @param unionFind Union-find structure to flatten - * @param smallRegions Output set of labels for small regions that need filling + * @brief Executes the BFS flood-fill algorithm to identify and fill bad data regions. + * @return Result indicating success or an error with a descriptive message. */ - static void phaseTwoGlobalResolution(ChunkAwareUnionFind& unionFind, std::unordered_set& smallRegions); - - /** - * @brief Phase 3: Relabel voxels based on region classification - * @param featureIdsStore Feature IDs data store - * @param cellPhasesPtr Cell phases array (could be null) - * @param provisionalLabels Map from voxel index to provisional label - * @param smallRegions Set of labels for small regions - * @param unionFind Union-find for looking up equivalences - * @param maxPhase Maximum phase value (for new phase assignment) - * @return Result<> invalid if a bulk read or write against the feature IDs store fails. - */ - Result<> phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, const std::unordered_map& provisionalLabels, - const std::unordered_set& smallRegions, ChunkAwareUnionFind& unionFind, size_t maxPhase) const; - - /** - * @brief Phase 4: Iterative morphological fill - * @param featureIdsStore Feature IDs data store - * @param dims Image geometry dimensions - * @param numFeatures Number of features - */ - void phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, size_t numFeatures) const; + Result<> operator()(); +private: DataStructure& m_DataStructure; const FillBadDataInputValues* m_InputValues = nullptr; const std::atomic_bool& m_ShouldCancel; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp new file mode 100644 index 0000000000..5f9f63b29c --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp @@ -0,0 +1,833 @@ +#include "FillBadDataCCL.hpp" + +#include "FillBadData.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/DataGroupUtilities.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" + +#include + +#include +#include + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +// FillBadData Algorithm Overview +// ----------------------------------------------------------------------------- +// +// This file implements an optimized algorithm for filling bad data (voxels with +// FeatureId == 0) in image geometries. The algorithm handles out-of-core datasets +// efficiently by processing data in Z-slice buffers and uses a four-phase approach: +// +// Phase 1: Z-Slice Sequential Connected Component Labeling (CCL) +// - Process Z-slices sequentially, assigning provisional labels to bad data regions +// - Use Union-Find to track equivalences between labels across slice boundaries +// - Track size of each connected component +// +// Phase 2: Global Resolution +// - Flatten Union-Find structure to resolve all equivalences +// - Accumulate region sizes to root labels +// +// Phase 3: Region Classification and Relabeling +// - Classify regions as "small" (below threshold) or "large" (above threshold) +// - Small regions: mark with -1 for filling in Phase 4 +// - Large regions: keep as 0 or assign to new phase (if requested) +// +// Phase 4: Iterative Morphological Fill (On-Disk Deferred) +// - Uses a temporary file to defer fills: Pass 1 writes (dest, src) pairs, +// Pass 2 reads them back and applies fills. +// - No O(N) memory allocations — uses O(features) vote counters + temp file I/O. +// +// ----------------------------------------------------------------------------- + +namespace +{ +// ----------------------------------------------------------------------------- +// Helper: Copy all components of a single tuple from src to dest in a data store. +// ----------------------------------------------------------------------------- +template +void copyTuple(AbstractDataStore& store, int64 dest, int64 src) +{ + const usize numComp = store.getNumberOfComponents(); + auto buffer = std::make_unique(numComp); + store.copyIntoBuffer(static_cast(src) * numComp, nonstd::span(buffer.get(), numComp)); + store.copyFromBuffer(static_cast(dest) * numComp, nonstd::span(buffer.get(), numComp)); +} + +// Functor for type-dispatched single-tuple copy +struct CopyTupleFunctor +{ + template + void operator()(IDataArray* dataArray, int64 dest, int64 src) + { + auto& store = dataArray->template getIDataStoreRefAs>(); + copyTuple(store, dest, src); + } +}; + +// Functor for type-dispatched slice-buffered copy of all pairs for one array. +// Uses a 3-slice rolling window to apply fills with bulk I/O. +struct SliceBufferedCopyFunctor +{ + template + void operator()(IDataArray* dataArray, const std::vector>& allPairs, usize pairsWritten, usize sliceTuples, int64 sliceStride, int64 dimZ) + { + if constexpr(std::is_same_v) + { + // std::vector doesn't support .data() — fall back to per-tuple copy. + // Bool cell arrays are extremely rare in practice. + auto& store = dataArray->template getIDataStoreRefAs>(); + for(usize pi = 0; pi < pairsWritten; pi++) + { + copyTuple(store, allPairs[pi][0], allPairs[pi][1]); + } + } + else + { + auto& store = dataArray->template getIDataStoreRefAs>(); + const usize numComp = store.getNumberOfComponents(); + const usize sliceValues = sliceTuples * numComp; + + std::vector buf(3 * sliceValues); + int64 curZ = -2; + + for(usize pi = 0; pi < pairsWritten; pi++) + { + const int64 dest = allPairs[pi][0]; + const int64 src = allPairs[pi][1]; + const int64 dz = dest / sliceStride; + + if(dz != curZ) + { + // Write back previous current slice + if(curZ >= 0) + { + store.copyFromBuffer(static_cast(curZ) * sliceValues, nonstd::span(buf.data() + sliceValues, sliceValues)); + } + curZ = dz; + if(curZ > 0) + { + store.copyIntoBuffer(static_cast(curZ - 1) * sliceValues, nonstd::span(buf.data(), sliceValues)); + } + store.copyIntoBuffer(static_cast(curZ) * sliceValues, nonstd::span(buf.data() + sliceValues, sliceValues)); + if(curZ + 1 < dimZ) + { + store.copyIntoBuffer(static_cast(curZ + 1) * sliceValues, nonstd::span(buf.data() + 2 * sliceValues, sliceValues)); + } + } + + const usize destInSlice = static_cast(dest - dz * sliceStride); + const int64 sz = src / sliceStride; + const usize srcInSlice = static_cast(src - sz * sliceStride); + const usize srcSlot = (sz == curZ - 1) ? 0 : (sz == curZ) ? 1 : 2; + + // Copy all components from src to dest + for(usize c = 0; c < numComp; c++) + { + buf[sliceValues + destInSlice * numComp + c] = buf[srcSlot * sliceValues + srcInSlice * numComp + c]; + } + } + // Write back final slice + if(curZ >= 0) + { + store.copyFromBuffer(static_cast(curZ) * sliceValues, nonstd::span(buf.data() + sliceValues, sliceValues)); + } + } + } +}; + +// RAII wrapper for std::FILE* that guarantees cleanup of the temporary file +// on destruction. This ensures the temp file is closed (and thus deleted by +// the OS, since std::tmpfile creates an anonymous file) even if Phase 4 +// returns early due to cancellation or error. Copy/assignment are deleted +// to enforce single-ownership semantics. +struct TempFileGuard +{ + std::FILE* file = nullptr; + + TempFileGuard() = default; + ~TempFileGuard() + { + if(file != nullptr) + { + std::fclose(file); + } + } + + TempFileGuard(const TempFileGuard&) = delete; + TempFileGuard& operator=(const TempFileGuard&) = delete; +}; +} // namespace + +// ----------------------------------------------------------------------------- +// FillBadData Implementation +// ----------------------------------------------------------------------------- + +FillBadDataCCL::FillBadDataCCL(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +FillBadDataCCL::~FillBadDataCCL() noexcept = default; + +// ----------------------------------------------------------------------------- +const std::atomic_bool& FillBadDataCCL::getCancel() const +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +// PHASE 1: Z-Slice Sequential Connected Component Labeling (CCL) +// ----------------------------------------------------------------------------- +// +// Performs connected component labeling on bad data voxels (FeatureId == 0) +// using a Z-slice sequential scanline algorithm. Uses positive labels and an +// in-memory provisional labels buffer to avoid cross-slice OOC reads. +// +// @param featureIdsStore The feature IDs data store (maybe out-of-core) +// @param unionFind Union-Find structure for tracking label equivalences +// @param nextLabel Next label to assign (incremented as new labels are created) +// @param dims Image dimensions [X, Y, Z] +// ----------------------------------------------------------------------------- +void FillBadDataCCL::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, UnionFind& unionFind, int32& nextLabel, const std::array& dims) +{ + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + + // Rolling 2-slice buffer for backward neighbor label reads. + // The scanline CCL algorithm only needs to look at three backward neighbors: + // x-1 (same slice), y-1 (same slice), and z-1 (previous slice). So we only + // need the current and immediately previous Z-slice labels in memory. The + // buffer alternates between even/odd Z indices via (z % 2) indexing. + // This gives O(dimX * dimY) memory instead of O(volume). + std::vector labelBuffer(2 * sliceSize, 0); + + // Temporary buffer for reading/writing featureIds one Z-slice at a time + std::vector featureIdsSlice(sliceSize); + + // Process each Z-slice sequentially + for(usize z = 0; z < static_cast(dims[2]); z++) + { + featureIdsStore.copyIntoBuffer(z * sliceSize, nonstd::span(featureIdsSlice.data(), sliceSize)); + + // Clear current slice in rolling label buffer for this z + const usize curOff = (z % 2) * sliceSize; + std::fill(labelBuffer.begin() + curOff, labelBuffer.begin() + curOff + sliceSize, 0); + const usize prevOff = ((z + 1) % 2) * sliceSize; + + for(usize y = 0; y < static_cast(dims[1]); y++) + { + for(usize x = 0; x < static_cast(dims[0]); x++) + { + const usize inSlice = y * static_cast(dims[0]) + x; + + // Only process bad data voxels (FeatureId == 0) + if(featureIdsSlice[inSlice] != 0) + { + continue; + } + + // Check backward neighbors using rolling buffer + int32 assignedLabel = 0; + + if(x > 0) + { + int32 neighLabel = labelBuffer[curOff + inSlice - 1]; + if(neighLabel > 0) + { + assignedLabel = neighLabel; + } + } + + if(y > 0) + { + int32 neighLabel = labelBuffer[curOff + inSlice - static_cast(dims[0])]; + if(neighLabel > 0) + { + if(assignedLabel == 0) + { + assignedLabel = neighLabel; + } + else if(assignedLabel != neighLabel) + { + unionFind.unite(assignedLabel, neighLabel); + } + } + } + + if(z > 0) + { + int32 neighLabel = labelBuffer[prevOff + inSlice]; + if(neighLabel > 0) + { + if(assignedLabel == 0) + { + assignedLabel = neighLabel; + } + else if(assignedLabel != neighLabel) + { + unionFind.unite(assignedLabel, neighLabel); + } + } + } + + if(assignedLabel == 0) + { + assignedLabel = nextLabel++; + unionFind.find(assignedLabel); + } + + // Write the provisional label to both the rolling buffer (for + // backward neighbor reads by subsequent voxels) and the featureIds + // slice buffer (persisted for Phases 2-3 to read back). + labelBuffer[curOff + inSlice] = assignedLabel; + featureIdsSlice[inSlice] = assignedLabel; + + // Accumulate region size: each voxel contributes 1 to its label. + // After Phase 2 flattening, sizes are aggregated to root labels + // so we can classify regions by total voxel count. + unionFind.addSize(assignedLabel, 1); + } + } + + featureIdsStore.copyFromBuffer(z * sliceSize, nonstd::span(featureIdsSlice.data(), sliceSize)); + } +} + +// ----------------------------------------------------------------------------- +// PHASE 2: Global Resolution of Equivalences +// ----------------------------------------------------------------------------- +void FillBadDataCCL::phaseTwoGlobalResolution(UnionFind& unionFind) +{ + unionFind.flatten(); +} + +// ----------------------------------------------------------------------------- +// PHASE 3: Region Classification and Relabeling +// ----------------------------------------------------------------------------- +// +// Classifies bad data regions as "small" or "large" based on size threshold: +// - Small regions (< minAllowedDefectSize): marked with -1 for filling in Phase 4 +// - Large regions (>= minAllowedDefectSize): kept as 0 (or assigned new phase) +// ----------------------------------------------------------------------------- +void FillBadDataCCL::phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, int32 startLabel, int32 nextLabel, UnionFind& unionFind, usize maxPhase) const +{ + const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); + const SizeVec3 udims = selectedImageGeom.getDimensions(); + + // Build a vector-based classification: isSmallRoot[label] = 1 if small, 0 if large. + // + // The startLabel boundary is critical: provisional CCL labels were assigned + // starting at (maxExistingFeatureId + 1) during Phase 1, so labels in the + // range [1, startLabel) are original good feature IDs that must NOT be + // touched. Only labels in [startLabel, nextLabel) are CCL-assigned bad-data + // region labels that need classification and relabeling. + std::vector isSmallRoot(static_cast(nextLabel), 0); + for(int32 label = startLabel; label < nextLabel; label++) + { + int64 root = unionFind.find(label); + if(root == label) + { + uint64 regionSize = unionFind.getSize(root); + if(regionSize < static_cast(m_InputValues->minAllowedDefectSizeValue)) + { + isSmallRoot[root] = 1; + } + } + } + + // Temporary buffer for reading/writing featureIds one Z-slice at a time + std::vector sliceData(static_cast(udims[0]) * static_cast(udims[1])); + const usize sliceSize = sliceData.size(); + + // Optional cellPhases buffer for bulk read/write (avoids per-element OOC access) + const bool needPhasesBuffer = m_InputValues->storeAsNewPhase && cellPhasesPtr != nullptr; + std::vector phasesSlice; + Int32AbstractDataStore* cellPhasesStorePtr = nullptr; + if(needPhasesBuffer) + { + phasesSlice.resize(sliceSize); + cellPhasesStorePtr = &cellPhasesPtr->getDataStoreRef(); + } + + // Read provisional labels from featureIds store (written during Phase 1) + // and relabel based on region classification. + // Only voxels with label >= startLabel are provisional CCL labels (bad data). + // Voxels with label in [1, startLabel) are original good feature IDs — leave them alone. + for(usize z = 0; z < udims[2]; z++) + { + featureIdsStore.copyIntoBuffer(z * sliceSize, nonstd::span(sliceData.data(), sliceSize)); + + bool phasesModified = false; + if(needPhasesBuffer) + { + cellPhasesStorePtr->copyIntoBuffer(z * sliceSize, nonstd::span(phasesSlice.data(), sliceSize)); + } + + for(usize y = 0; y < udims[1]; y++) + { + for(usize x = 0; x < udims[0]; x++) + { + const usize inSlice = y * udims[0] + x; + + int32 label = sliceData[inSlice]; + if(label >= startLabel) + { + int64 root = unionFind.find(label); + + if(isSmallRoot[root] != 0) + { + sliceData[inSlice] = -1; + } + else + { + sliceData[inSlice] = 0; + + if(needPhasesBuffer) + { + phasesSlice[inSlice] = static_cast(maxPhase) + 1; + phasesModified = true; + } + } + } + } + } + + featureIdsStore.copyFromBuffer(z * sliceSize, nonstd::span(sliceData.data(), sliceSize)); + if(phasesModified) + { + cellPhasesStorePtr->copyFromBuffer(z * sliceSize, nonstd::span(phasesSlice.data(), sliceSize)); + } + } +} + +// ----------------------------------------------------------------------------- +// PHASE 4: Iterative Morphological Fill (On-Disk Deferred) +// ----------------------------------------------------------------------------- +// +// Uses a temporary file to avoid O(N) memory allocations. Each iteration: +// Pass 1 (Vote): Scan voxels using a 3-slice rolling window. For each -1 voxel, +// find the best positive-featureId neighbor via majority vote. Write (dest, src) +// pairs to a temp file. featureIds is read-only during this pass. +// Pass 2 (Apply): Read pairs back from the temp file. Copy all cell data array +// components from src to dest. Update featureIds last. +// ----------------------------------------------------------------------------- +Result<> FillBadDataCCL::phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, usize numFeatures) const +{ + const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); + + // Feature vote counter: O(features) not O(voxels) + std::vector featureNumber(numFeatures + 1, 0); + + // Get cell arrays that need updating during filling + std::optional> allChildArrays = GetAllChildDataPaths(m_DataStructure, selectedImageGeom.getCellDataPath(), DataObject::Type::DataArray, m_InputValues->ignoredDataArrayPaths); + std::vector voxelArrayNames; + if(allChildArrays.has_value()) + { + voxelArrayNames = allChildArrays.value(); + } + + // Open a temporary file for deferred fill pairs. We use a temp file instead + // of an O(N) in-memory neighbors vector so that Phase 4 stays OOC-friendly. + // Pass 1 writes (dest, src) index pairs to the file; Pass 2 reads them back + // and applies the fills. This two-pass approach ensures that featureIds are + // read-only during the vote scan (Pass 1), so all votes see the pre-iteration + // state. The TempFileGuard RAII wrapper guarantees the file is closed even + // if an early return or error occurs, preventing temp file leaks. + TempFileGuard tmpGuard; + tmpGuard.file = std::tmpfile(); + if(tmpGuard.file == nullptr) + { + return MakeErrorResult(-87010, "Phase 4/4: Failed to create temporary file for deferred fill"); + } + + usize count = 1; + usize iteration = 0; + usize pairsWritten = 0; + + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + + while(count != 0) + { + iteration++; + count = 0; + + // Rewind for this iteration's writes + std::rewind(tmpGuard.file); + pairsWritten = 0; + + // Pass 1 (Vote): Z-slice scan using a 3-slice rolling window, writing + // (dest, src) pairs to temp file. featureIds is read-only during this + // pass — two-pass semantics are automatic. + std::vector prevSlice(sliceSize, 0); + std::vector curSlice(sliceSize); + std::vector nextSlice(sliceSize, 0); + + featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + if(dims[2] > 1) + { + featureIdsStore.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + + for(int64 z = 0; z < dims[2]; z++) + { + if(m_ShouldCancel) + { + return {}; + } + + for(int64 y = 0; y < dims[1]; y++) + { + for(int64 x = 0; x < dims[0]; x++) + { + const usize inSlice = static_cast(y) * static_cast(dims[0]) + static_cast(x); + int32 featureName = curSlice[inSlice]; + + if(featureName < 0) + { + count++; + int32 most = 0; + int64 bestNeighbor = -1; + + // Check 6 face neighbors using the 3-slice window + // -X neighbor + if(x > 0) + { + int32 feature = curSlice[inSlice - 1]; + if(feature > 0) + { + featureNumber[feature]++; + if(featureNumber[feature] > most) + { + most = featureNumber[feature]; + bestNeighbor = z * dims[0] * dims[1] + y * dims[0] + (x - 1); + } + } + } + // +X neighbor + if(x < dims[0] - 1) + { + int32 feature = curSlice[inSlice + 1]; + if(feature > 0) + { + featureNumber[feature]++; + if(featureNumber[feature] > most) + { + most = featureNumber[feature]; + bestNeighbor = z * dims[0] * dims[1] + y * dims[0] + (x + 1); + } + } + } + // -Y neighbor + if(y > 0) + { + int32 feature = curSlice[inSlice - static_cast(dims[0])]; + if(feature > 0) + { + featureNumber[feature]++; + if(featureNumber[feature] > most) + { + most = featureNumber[feature]; + bestNeighbor = z * dims[0] * dims[1] + (y - 1) * dims[0] + x; + } + } + } + // +Y neighbor + if(y < dims[1] - 1) + { + int32 feature = curSlice[inSlice + static_cast(dims[0])]; + if(feature > 0) + { + featureNumber[feature]++; + if(featureNumber[feature] > most) + { + most = featureNumber[feature]; + bestNeighbor = z * dims[0] * dims[1] + (y + 1) * dims[0] + x; + } + } + } + // -Z neighbor + if(z > 0) + { + int32 feature = prevSlice[inSlice]; + if(feature > 0) + { + featureNumber[feature]++; + if(featureNumber[feature] > most) + { + most = featureNumber[feature]; + bestNeighbor = (z - 1) * dims[0] * dims[1] + y * dims[0] + x; + } + } + } + // +Z neighbor + if(z < dims[2] - 1) + { + int32 feature = nextSlice[inSlice]; + if(feature > 0) + { + featureNumber[feature]++; + if(featureNumber[feature] > most) + { + most = featureNumber[feature]; + bestNeighbor = (z + 1) * dims[0] * dims[1] + y * dims[0] + x; + } + } + } + + // Reset vote counters by re-visiting only the neighbors that + // were actually incremented above. This sets featureNumber[feature] + // back to 0 for each neighbor's feature, avoiding the need to zero + // the entire featureNumber vector (which would be O(numFeatures) + // per voxel). Since at most 6 neighbors are visited, this reset + // is O(1) per voxel. + if(x > 0) + { + int32 f = curSlice[inSlice - 1]; + if(f > 0) + { + featureNumber[f] = 0; + } + } + if(x < dims[0] - 1) + { + int32 f = curSlice[inSlice + 1]; + if(f > 0) + { + featureNumber[f] = 0; + } + } + if(y > 0) + { + int32 f = curSlice[inSlice - static_cast(dims[0])]; + if(f > 0) + { + featureNumber[f] = 0; + } + } + if(y < dims[1] - 1) + { + int32 f = curSlice[inSlice + static_cast(dims[0])]; + if(f > 0) + { + featureNumber[f] = 0; + } + } + if(z > 0) + { + int32 f = prevSlice[inSlice]; + if(f > 0) + { + featureNumber[f] = 0; + } + } + if(z < dims[2] - 1) + { + int32 f = nextSlice[inSlice]; + if(f > 0) + { + featureNumber[f] = 0; + } + } + + // Write (dest, src) pair to temp file if a valid neighbor was found + if(bestNeighbor >= 0) + { + std::array pair = {z * dims[0] * dims[1] + y * dims[0] + x, bestNeighbor}; + if(std::fwrite(pair.data(), sizeof(int64), 2, tmpGuard.file) != 2) + { + return MakeErrorResult(-87012, "Phase 4/4: Failed to write fill pair to temporary file"); + } + pairsWritten++; + } + } + } + } + + // Shift 3-slice window forward + std::swap(prevSlice, curSlice); + std::swap(curSlice, nextSlice); + if(z + 2 < dims[2]) + { + featureIdsStore.copyIntoBuffer(static_cast(z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); + } + } + + if(count == 0) + { + break; + } + + // Pass 2 (Apply): Read all pairs from temp file, then apply fills using + // slice-buffered I/O per array. Pairs are in Z-Y-X order, so each array + // is processed with a 3-slice rolling window. This converts millions of + // per-tuple OOC accesses into ~600 bulk slice reads/writes per array. + + // Read all pairs into memory (788K pairs × 16 bytes = ~12 MB — acceptable) + std::vector> allPairs(pairsWritten); + std::rewind(tmpGuard.file); + for(usize i = 0; i < pairsWritten; i++) + { + std::fread(allPairs[i].data(), sizeof(int64), 2, tmpGuard.file); + } + + const int64 sliceStride = dims[0] * dims[1]; + const usize sliceTuples = sliceSize; + + // Apply featureIds fills using 3-slice buffer + { + std::vector fidBuf(3 * sliceTuples, 0); + int64 curZ = -2; + + for(usize pi = 0; pi < pairsWritten; pi++) + { + const int64 dest = allPairs[pi][0]; + const int64 src = allPairs[pi][1]; + const int64 dz = dest / sliceStride; + + if(dz != curZ) + { + // Write back previous current slice + if(curZ >= 0) + { + featureIdsStore.copyFromBuffer(static_cast(curZ) * sliceTuples, nonstd::span(fidBuf.data() + sliceTuples, sliceTuples)); + } + curZ = dz; + // Load 3-slice window: [prev | cur | next] + if(curZ > 0) + { + featureIdsStore.copyIntoBuffer(static_cast(curZ - 1) * sliceTuples, nonstd::span(fidBuf.data(), sliceTuples)); + } + featureIdsStore.copyIntoBuffer(static_cast(curZ) * sliceTuples, nonstd::span(fidBuf.data() + sliceTuples, sliceTuples)); + if(curZ + 1 < dims[2]) + { + featureIdsStore.copyIntoBuffer(static_cast(curZ + 1) * sliceTuples, nonstd::span(fidBuf.data() + 2 * sliceTuples, sliceTuples)); + } + } + + const usize destInSlice = static_cast(dest - dz * sliceStride); + const int64 sz = src / sliceStride; + const usize srcInSlice = static_cast(src - sz * sliceStride); + const usize srcSlot = (sz == curZ - 1) ? 0 : (sz == curZ) ? 1 : 2; + + fidBuf[sliceTuples + destInSlice] = fidBuf[srcSlot * sliceTuples + srcInSlice]; + } + // Write back final slice + if(curZ >= 0) + { + featureIdsStore.copyFromBuffer(static_cast(curZ) * sliceTuples, nonstd::span(fidBuf.data() + sliceTuples, sliceTuples)); + } + } + + // Apply non-featureIds cell array fills, one array at a time + for(const auto& cellArrayPath : voxelArrayNames) + { + if(cellArrayPath == m_InputValues->featureIdsArrayPath) + { + continue; + } + auto* cellArray = m_DataStructure.getDataAs(cellArrayPath); + // Type-dispatched slice-buffered fill + ExecuteDataFunction(SliceBufferedCopyFunctor{}, cellArray->getDataType(), cellArray, allPairs, pairsWritten, sliceTuples, sliceStride, dims[2]); + } + + featureIdsStore.flush(); + } + + m_MessageHandler({IFilter::Message::Type::Info, fmt::format(" Completed in {} iteration{}", iteration, iteration == 1 ? "" : "s")}); + return {}; +} + +// ----------------------------------------------------------------------------- +// Main Algorithm Entry Point +// ----------------------------------------------------------------------------- +Result<> FillBadDataCCL::operator()() +{ + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath)->getDataStoreRef(); + const auto& selectedImageGeom = m_DataStructure.getDataRefAs(m_InputValues->inputImageGeometry); + const SizeVec3 udims = selectedImageGeom.getDimensions(); + + std::array dims = { + static_cast(udims[0]), + static_cast(udims[1]), + static_cast(udims[2]), + }; + + const usize totalPoints = featureIdsStore.getNumberOfTuples(); + + // Get cell phases array if we need to assign large regions to a new phase + Int32Array* cellPhasesPtr = nullptr; + usize maxPhase = 0; + + if(m_InputValues->storeAsNewPhase) + { + cellPhasesPtr = m_DataStructure.getDataAs(m_InputValues->cellPhasesArrayPath); + } + + // Chunked scan: find max feature ID and optionally max phase using bulk reads + usize numFeatures = 0; + const usize k_ScanBatchSize = static_cast(dims[0]) * static_cast(dims[1]); + std::vector scanBuffer(k_ScanBatchSize); + for(usize offset = 0; offset < totalPoints; offset += k_ScanBatchSize) + { + const usize batchSize = std::min(k_ScanBatchSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(scanBuffer.data(), batchSize)); + for(usize i = 0; i < batchSize; i++) + { + if(scanBuffer[i] > static_cast(numFeatures)) + { + numFeatures = scanBuffer[i]; + } + } + } + // Separate chunked scan for cellPhases if needed + if(cellPhasesPtr != nullptr) + { + auto& cellPhasesStore = cellPhasesPtr->getDataStoreRef(); + for(usize offset = 0; offset < totalPoints; offset += k_ScanBatchSize) + { + const usize batchSize = std::min(k_ScanBatchSize, totalPoints - offset); + cellPhasesStore.copyIntoBuffer(offset, nonstd::span(scanBuffer.data(), batchSize)); + for(usize i = 0; i < batchSize; i++) + { + if(static_cast(scanBuffer[i]) > maxPhase) + { + maxPhase = scanBuffer[i]; + } + } + } + } + + // Initialize data structures for connected component labeling. + // Start provisional labels AFTER the max existing feature ID to avoid collisions. + // Existing feature IDs are in [1, numFeatures], so provisional labels start at numFeatures+1. + UnionFind unionFind; + const int32 startLabel = static_cast(numFeatures) + 1; + int32 nextLabel = startLabel; + + // Phase 1: Z-Slice Sequential Connected Component Labeling + // Uses a 2-slice rolling buffer (O(slice) memory) for backward neighbor reads. + // Writes provisional labels to featureIds store for Phases 2-3. + m_MessageHandler({IFilter::Message::Type::Info, "Phase 1/4: Labeling connected components..."}); + phaseOneCCL(featureIdsStore, unionFind, nextLabel, dims); + + // Phase 2: Global Resolution of equivalences + m_MessageHandler({IFilter::Message::Type::Info, "Phase 2/4: Resolving region equivalences..."}); + phaseTwoGlobalResolution(unionFind); + + // Phase 3: Relabeling based on region size classification + // Reads provisional labels from featureIds store (written during Phase 1) + m_MessageHandler({IFilter::Message::Type::Info, "Phase 3/4: Classifying region sizes..."}); + phaseThreeRelabeling(featureIdsStore, cellPhasesPtr, startLabel, nextLabel, unionFind, maxPhase); + + // Phase 4: Iterative morphological fill + m_MessageHandler({IFilter::Message::Type::Info, "Phase 4/4: Filling small defects..."}); + auto result = phaseFourIterativeFill(featureIdsStore, dims, numFeatures); + return result; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp new file mode 100644 index 0000000000..4822e180a7 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp @@ -0,0 +1,76 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Utilities/UnionFind.hpp" + +namespace nx::core +{ + +// Forward declarations +template +class DataArray; +using Int32Array = DataArray; + +template +class AbstractDataStore; +using Int32AbstractDataStore = AbstractDataStore; + +struct FillBadDataInputValues; + +/** + * @class FillBadDataCCL + * @brief CCL-based algorithm for filling bad data regions, optimized for out-of-core. + * + * Uses chunk-sequential connected component labeling with a 2-slice rolling buffer + * to avoid O(N) memory allocations. Designed for datasets that may exceed available RAM. + * + * @see FillBadDataBFS for the in-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + */ +class SIMPLNXCORE_EXPORT FillBadDataCCL +{ +public: + /** + * @brief Constructs the CCL fill algorithm with the required context. + * @param dataStructure The data structure containing the arrays to process. + * @param mesgHandler Handler for progress and informational messages. + * @param shouldCancel Cancellation flag checked during execution. + * @param inputValues Filter parameter values controlling fill behavior. + */ + FillBadDataCCL(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues); + ~FillBadDataCCL() noexcept; + + FillBadDataCCL(const FillBadDataCCL&) = delete; + FillBadDataCCL(FillBadDataCCL&&) noexcept = delete; + FillBadDataCCL& operator=(const FillBadDataCCL&) = delete; + FillBadDataCCL& operator=(FillBadDataCCL&&) noexcept = delete; + + /** + * @brief Executes the CCL-based algorithm to identify and fill bad data regions. + * @return Result indicating success or an error with a descriptive message. + */ + Result<> operator()(); + + /** + * @brief Returns the cancellation flag reference. + * @return Reference to the atomic cancellation flag. + */ + const std::atomic_bool& getCancel() const; + +private: + static void phaseOneCCL(Int32AbstractDataStore& featureIdsStore, UnionFind& unionFind, int32& nextLabel, const std::array& dims); + static void phaseTwoGlobalResolution(UnionFind& unionFind); + void phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, int32 startLabel, int32 nextLabel, UnionFind& unionFind, usize maxPhase) const; + Result<> phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, usize numFeatures) const; + + DataStructure& m_DataStructure; + const FillBadDataInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp new file mode 100644 index 0000000000..624d7b60b2 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp @@ -0,0 +1,29 @@ +#include "IdentifySample.hpp" + +#include "IdentifySampleBFS.hpp" +#include "IdentifySampleCCL.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +IdentifySample::IdentifySample(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +IdentifySample::~IdentifySample() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> IdentifySample::operator()() +{ + auto* maskArray = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); + + return DispatchAlgorithm({maskArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp new file mode 100644 index 0000000000..a24a219f25 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp @@ -0,0 +1,50 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ArraySelectionParameter.hpp" +#include "simplnx/Parameters/BoolParameter.hpp" +#include "simplnx/Parameters/ChoicesParameter.hpp" +#include "simplnx/Parameters/GeometrySelectionParameter.hpp" + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT IdentifySampleInputValues +{ + BoolParameter::ValueType FillHoles; + GeometrySelectionParameter::ValueType InputImageGeometryPath; + ArraySelectionParameter::ValueType MaskArrayPath; + BoolParameter::ValueType SliceBySlice; + ChoicesParameter::ValueType SliceBySlicePlaneIndex; +}; + +/** + * @class IdentifySample + * @brief This algorithm implements support code for the IdentifySampleFilter + */ + +class SIMPLNXCORE_EXPORT IdentifySample +{ +public: + IdentifySample(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues); + ~IdentifySample() noexcept; + + IdentifySample(const IdentifySample&) = delete; + IdentifySample(IdentifySample&&) noexcept = delete; + IdentifySample& operator=(const IdentifySample&) = delete; + IdentifySample& operator=(IdentifySample&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const IdentifySampleInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp index 954016d883..43eaa152a0 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp @@ -1,63 +1,92 @@ #include "IdentifySampleBFS.hpp" +#include "IdentifySample.hpp" +#include "IdentifySampleCommon.hpp" + #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" +#include "simplnx/Utilities/MessageHelper.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" +#include + using namespace nx::core; namespace { +// ============================================================================= +// IdentifySampleBFSFunctor +// ============================================================================= +// BFS flood-fill algorithm for identifying the largest connected component of +// "good" voxels in an image geometry, then optionally filling interior holes. +// +// The algorithm has two phases: +// +// Phase 1 (Find Largest Component): +// BFS flood-fill discovers all connected components of good voxels +// (goodVoxels == true). Each component is found by starting BFS from an +// unchecked good voxel and expanding to all face-adjacent good neighbors. +// The largest component by voxel count is tracked as "the sample". After +// all components are found, any good voxels NOT in the largest component +// are set to false (they are noise or satellite regions). +// Uses O(N) memory: checked + sample vectors (std::vector, 1 bit each). +// +// Phase 2 (Hole Fill, optional): +// If fillHoles is true, a second BFS pass runs on bad voxels +// (goodVoxels == false). Each connected component of bad voxels is +// discovered via BFS. During BFS, a `touchesBoundary` flag tracks whether +// any voxel in the component lies on the domain boundary (x/y/z == 0 or +// max). If the component does NOT touch the boundary, it is fully enclosed +// by the sample and is an interior hole -- all its voxels are set to true. +// If it touches the boundary, it is external empty space and left as-is. +// +// NOTE: Uses std::vector (1 bit per voxel) for minimal memory overhead. +// Fast for in-core data where random access is O(1), but causes chunk +// thrashing in OOC mode due to BFS visiting neighbors across chunk boundaries. +// Use IdentifySampleCCL for out-of-core compatible processing. +// ============================================================================= struct IdentifySampleBFSFunctor { template void operator()(const ImageGeom* imageGeom, IDataArray* goodVoxelsPtr, bool fillHoles, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel) { - ShapeType cDims = {1}; auto& goodVoxels = goodVoxelsPtr->template getIDataStoreRefAs>(); const auto totalPoints = static_cast(goodVoxelsPtr->getNumberOfTuples()); SizeVec3 udims = imageGeom->getDimensions(); - std::array dims = { - static_cast(udims[0]), - static_cast(udims[1]), - static_cast(udims[2]), + std::array dims = { + static_cast(udims[0]), + static_cast(udims[1]), + static_cast(udims[2]), }; - int64_t neighborPoint = 0; + int64 neighborPoint = 0; std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); std::vector currentVList; - std::vector checked(totalPoints, false); - std::vector sample(totalPoints, false); + std::vector checked(totalPoints, false); // O(N) bits: tracks visited voxels + std::vector sample(totalPoints, false); // O(N) bits: marks voxels in the largest component int64 biggestBlock = 0; - // In this loop over the data we are finding the biggest contiguous set of GoodVoxels and calling that the 'sample' All GoodVoxels that do not touch the 'sample' - // are flipped to be called 'bad' voxels or 'not sample' - float threshold = 0.0f; + MessageHelper messageHelper(messageHandler); + + // --- Phase 1: Find the largest contiguous set of good voxels ------------ + // BFS flood-fill from each unvisited good voxel. Track the largest + // connected component found so far. + messageHelper.sendMessage("Phase 1: Finding largest connected component of good voxels..."); for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) { if(shouldCancel) { return; } - const float percentIncrement = static_cast(voxelIndex) / static_cast(totalPoints) * 100.0f; - if(percentIncrement > threshold) - { - messageHandler(IFilter::Message::Type::Info, fmt::format("Completed: {}", percentIncrement)); - threshold = threshold + 5.0f; - if(threshold < percentIncrement) - { - threshold = percentIncrement; - } - } - if(!checked[voxelIndex] && goodVoxels.getValue(voxelIndex)) { + // Start BFS from this seed voxel to discover one connected component currentVList.push_back(voxelIndex); usize count = 0; while(count < currentVList.size()) @@ -83,6 +112,7 @@ struct IdentifySampleBFSFunctor } count++; } + // If this component is the largest found so far, record it as the sample if(static_cast(currentVList.size()) >= biggestBlock) { biggestBlock = currentVList.size(); @@ -95,6 +125,8 @@ struct IdentifySampleBFSFunctor currentVList.clear(); } } + // Any good voxels NOT in the largest component are noise/satellites -- + // set them to false so only the primary sample remains. for(int64 i = 0; i < totalPoints; i++) { if(!sample[i] && goodVoxels.getValue(i)) @@ -105,12 +137,16 @@ struct IdentifySampleBFSFunctor sample.clear(); checked.assign(totalPoints, false); - // In this loop we are going to 'close' all the 'holes' inside the region already identified as the 'sample' if the user chose to do so. - // This is done by flipping all 'bad' voxel features that do not touch the outside of the sample (i.e. they are fully contained inside the 'sample'). - threshold = 0.0F; + // --- Phase 2: Hole fill (optional) ---------------------------------------- + // BFS on bad voxels (goodVoxels == false). Each connected component of + // bad voxels is checked: if any voxel in the component touches a domain + // boundary face (x/y/z == 0 or max), the component is external empty + // space and is left as-is. If the component is fully enclosed by the + // sample (touchesBoundary == false), it is an interior hole and all + // its voxels are set to true. if(fillHoles) { - messageHandler(IFilter::Message::Type::Info, fmt::format("Filling holes in sample...")); + messageHelper.sendMessage("Phase 2: Filling holes in sample..."); bool touchesBoundary = false; for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) @@ -119,18 +155,11 @@ struct IdentifySampleBFSFunctor { return; } - const float percentIncrement = static_cast(voxelIndex) / static_cast(totalPoints) * 100.0f; - if(percentIncrement > threshold) - { - threshold = threshold + 5.0f; - if(threshold < percentIncrement) - { - threshold = percentIncrement; - } - } - if(!checked[voxelIndex] && !goodVoxels.getValue(voxelIndex)) { + // BFS from this bad voxel to discover one connected component of + // bad data. Track whether any voxel in the component is on a + // domain boundary face. currentVList.push_back(voxelIndex); usize count = 0; touchesBoundary = false; @@ -140,11 +169,11 @@ struct IdentifySampleBFSFunctor int64 xIdx = index % dims[0]; int64 yIdx = (index / dims[0]) % dims[1]; int64 zIdx = index / (dims[0] * dims[1]); + // Check if this voxel lies on any domain boundary face if(xIdx == 0 || xIdx == (dims[0] - 1) || yIdx == 0 || yIdx == (dims[1] - 1) || zIdx == 0 || zIdx == (dims[2] - 1)) { touchesBoundary = true; } - // Loop over the 6 face neighbors of the voxel std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); for(const auto& faceIndex : faceNeighborInternalIdx) { @@ -162,9 +191,11 @@ struct IdentifySampleBFSFunctor } count++; } + // If this bad-data component does not touch any boundary, it is + // an interior hole -- fill it by setting all voxels to true. if(!touchesBoundary) { - for(int64_t j : currentVList) + for(int64 j : currentVList) { goodVoxels.setValue(j, true); } @@ -176,221 +207,10 @@ struct IdentifySampleBFSFunctor checked.clear(); } }; - -struct IdentifySampleBFSSliceBySliceFunctor -{ - enum class Plane - { - XY, - XZ, - YZ - }; - - template - void operator()(const ImageGeom* imageGeom, IDataArray* goodVoxelsPtr, bool fillHoles, Plane plane, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel) - { - auto& goodVoxels = goodVoxelsPtr->template getIDataStoreRefAs>(); - - SizeVec3 uDims = imageGeom->getDimensions(); - const int64 dimX = static_cast(uDims[0]); - const int64 dimY = static_cast(uDims[1]); - const int64 dimZ = static_cast(uDims[2]); - - int64 planeDim1, planeDim2, fixedDim; - int64 stride1, stride2, fixedStride; - - switch(plane) - { - case Plane::XY: - planeDim1 = dimX; - planeDim2 = dimY; - fixedDim = dimZ; - stride1 = 1; - stride2 = dimX; - fixedStride = dimX * dimY; - break; - - case Plane::XZ: - planeDim1 = dimX; - planeDim2 = dimZ; - fixedDim = dimY; - stride1 = 1; - stride2 = dimX * dimY; - fixedStride = dimX; - break; - - case Plane::YZ: - planeDim1 = dimY; - planeDim2 = dimZ; - fixedDim = dimX; - stride1 = dimX; - stride2 = dimX * dimY; - fixedStride = 1; - break; - } - - for(int64 fixedIdx = 0; fixedIdx < fixedDim; ++fixedIdx) // Process each slice - { - if(shouldCancel) - { - return; - } - messageHandler(IFilter::Message::Type::Info, fmt::format("Slice {}", fixedIdx)); - - std::vector checked(planeDim1 * planeDim2, false); - std::vector sample(planeDim1 * planeDim2, false); - std::vector currentVList; - int64 biggestBlock = 0; - - // Identify the largest contiguous set of good voxels in the slice - for(int64 p2 = 0; p2 < planeDim2; ++p2) - { - for(int64 p1 = 0; p1 < planeDim1; ++p1) - { - int64 planeIndex = p2 * planeDim1 + p1; - int64 globalIndex = fixedIdx * fixedStride + p2 * stride2 + p1 * stride1; - - if(!checked[planeIndex] && goodVoxels.getValue(globalIndex)) - { - currentVList.push_back(planeIndex); - int64 count = 0; - - while(count < currentVList.size()) - { - int64 localIdx = currentVList[count]; - int64 localP1 = localIdx % planeDim1; - int64 localP2 = localIdx / planeDim1; - - for(int j = 0; j < 4; ++j) - { - int64 dp1[4] = {0, 0, -1, 1}; - int64 dp2[4] = {-1, 1, 0, 0}; - - int64 neighborP1 = localP1 + dp1[j]; - int64 neighborP2 = localP2 + dp2[j]; - - if(neighborP1 >= 0 && neighborP1 < planeDim1 && neighborP2 >= 0 && neighborP2 < planeDim2) - { - int64 neighborIdx = neighborP2 * planeDim1 + neighborP1; - int64 globalNeighborIdx = fixedIdx * fixedStride + neighborP2 * stride2 + neighborP1 * stride1; - - if(!checked[neighborIdx] && goodVoxels.getValue(globalNeighborIdx)) - { - currentVList.push_back(neighborIdx); - checked[neighborIdx] = true; - } - } - } - count++; - } - - if(static_cast(currentVList.size()) > biggestBlock) - { - biggestBlock = currentVList.size(); - sample.assign(planeDim1 * planeDim2, false); - for(int64 idx : currentVList) - { - sample[idx] = true; - } - } - currentVList.clear(); - } - } - } - if(shouldCancel) - { - return; - } - - for(int64 p2 = 0; p2 < planeDim2; ++p2) - { - for(int64 p1 = 0; p1 < planeDim1; ++p1) - { - int64 planeIndex = p2 * planeDim1 + p1; - int64 globalIndex = fixedIdx * fixedStride + p2 * stride2 + p1 * stride1; - - if(!sample[planeIndex]) - { - goodVoxels.setValue(globalIndex, false); - } - } - } - if(shouldCancel) - { - return; - } - - checked.assign(planeDim1 * planeDim2, false); - if(fillHoles) - { - for(int64 p2 = 0; p2 < planeDim2; ++p2) - { - for(int64 p1 = 0; p1 < planeDim1; ++p1) - { - int64 planeIndex = p2 * planeDim1 + p1; - int64 globalIndex = fixedIdx * fixedStride + p2 * stride2 + p1 * stride1; - - if(!checked[planeIndex] && !goodVoxels.getValue(globalIndex)) - { - currentVList.push_back(planeIndex); - int64 count = 0; - bool touchesBoundary = false; - - while(count < currentVList.size()) - { - int64 localIdx = currentVList[count]; - int64 localP1 = localIdx % planeDim1; - int64 localP2 = localIdx / planeDim1; - - if(localP1 == 0 || localP1 == planeDim1 - 1 || localP2 == 0 || localP2 == planeDim2 - 1) - { - touchesBoundary = true; - } - - for(int j = 0; j < 4; ++j) - { - int64 dp1[4] = {0, 0, -1, 1}; - int64 dp2[4] = {-1, 1, 0, 0}; - - int64 neighborP1 = localP1 + dp1[j]; - int64 neighborP2 = localP2 + dp2[j]; - - if(neighborP1 >= 0 && neighborP1 < planeDim1 && neighborP2 >= 0 && neighborP2 < planeDim2) - { - int64 neighborIdx = neighborP2 * planeDim1 + neighborP1; - int64 globalNeighborIdx = fixedIdx * fixedStride + neighborP2 * stride2 + neighborP1 * stride1; - - if(!checked[neighborIdx] && !goodVoxels.getValue(globalNeighborIdx)) - { - currentVList.push_back(neighborIdx); - checked[neighborIdx] = true; - } - } - } - count++; - } - - if(!touchesBoundary) - { - for(int64 idx : currentVList) - { - int64 globalP1 = idx % planeDim1; - int64 globalP2 = idx / planeDim1; - goodVoxels.setValue(fixedIdx * fixedStride + globalP2 * stride2 + globalP1 * stride1, true); - } - } - currentVList.clear(); - } - } - } - } - } - } -}; } // namespace // ----------------------------------------------------------------------------- -IdentifySampleBFS::IdentifySampleBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues) +IdentifySampleBFS::IdentifySampleBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const IdentifySampleInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -409,8 +229,8 @@ Result<> IdentifySampleBFS::operator()() if(m_InputValues->SliceBySlice) { - ExecuteDataFunction(IdentifySampleBFSSliceBySliceFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, - static_cast(m_InputValues->SliceBySlicePlaneIndex), m_MessageHandler, m_ShouldCancel); + ExecuteDataFunction(IdentifySampleSliceBySliceFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, + static_cast(m_InputValues->SliceBySlicePlaneIndex), m_MessageHandler, m_ShouldCancel); } else { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp index 57a8f93553..e506bcb158 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp @@ -5,32 +5,35 @@ #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ArraySelectionParameter.hpp" -#include "simplnx/Parameters/BoolParameter.hpp" -#include "simplnx/Parameters/ChoicesParameter.hpp" -#include "simplnx/Parameters/GeometrySelectionParameter.hpp" namespace nx::core { -struct SIMPLNXCORE_EXPORT IdentifySampleInputValues -{ - BoolParameter::ValueType FillHoles; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - ArraySelectionParameter::ValueType MaskArrayPath; - BoolParameter::ValueType SliceBySlice; - ChoicesParameter::ValueType SliceBySlicePlaneIndex; -}; +struct IdentifySampleInputValues; /** * @class IdentifySampleBFS - * @brief This algorithm implements support code for the IdentifySampleFilter + * @brief BFS flood-fill algorithm for identifying the largest sample region. + * + * This is the in-core-optimized implementation. It uses BFS (breadth-first search) + * with std::vector for tracking visited voxels, which is memory-efficient + * (1 bit per voxel) and fast when data is in contiguous memory. However, the random + * access pattern of BFS causes severe chunk thrashing in out-of-core mode. + * + * @see IdentifySampleCCL for the out-of-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ - class SIMPLNXCORE_EXPORT IdentifySampleBFS { public: - IdentifySampleBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues); + /** + * @brief Constructs the BFS sample identification algorithm with the required context. + * @param dataStructure The data structure containing the arrays to process. + * @param mesgHandler Handler for progress and informational messages. + * @param shouldCancel Cancellation flag checked during execution. + * @param inputValues Filter parameter values controlling identification behavior. + */ + IdentifySampleBFS(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const IdentifySampleInputValues* inputValues); ~IdentifySampleBFS() noexcept; IdentifySampleBFS(const IdentifySampleBFS&) = delete; @@ -38,6 +41,10 @@ class SIMPLNXCORE_EXPORT IdentifySampleBFS IdentifySampleBFS& operator=(const IdentifySampleBFS&) = delete; IdentifySampleBFS& operator=(IdentifySampleBFS&&) noexcept = delete; + /** + * @brief Executes the BFS flood-fill algorithm to identify the largest sample region. + * @return Result indicating success or an error with a descriptive message. + */ Result<> operator()(); private: diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp new file mode 100644 index 0000000000..454ef74f7b --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp @@ -0,0 +1,453 @@ +#include "IdentifySampleCCL.hpp" + +#include "IdentifySample.hpp" +#include "IdentifySampleCommon.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" + +#include +#include + +using namespace nx::core; + +namespace +{ +// ============================================================================= +// runForwardCCL +// ============================================================================= +// Generic Z-slice-sequential Connected Component Labeling function that works +// on any boolean condition. It processes the volume one Z-slice at a time using +// copyIntoBuffer for OOC-friendly reads, and a rolling 2-slice label buffer +// instead of storing labels for the entire volume. +// +// How it works: +// - Scans voxels in Z-slice order (z, y, x innermost). For each voxel where +// `condition(sliceData, inSlice)` returns true, checks three backward +// neighbors (x-1, y-1, z-1) for existing labels. +// - If no labeled neighbor exists, assigns a new provisional label. +// - If multiple differently-labeled neighbors exist, unites them in the +// union-find structure. +// - Tracks per-label voxel counts (labelSizes) so the largest root can be +// identified after flattening, without a separate counting pass. +// +// The `condition` lambda determines which voxels to label. For example: +// - `data[inSlice] == true` labels good voxels (sample identification) +// - `!data[inSlice]` labels bad voxels (hole detection) +// +// Returns a CCLResult containing the union-find, accumulated root sizes, +// the next available label, and the largest root/size. +// ============================================================================= +struct CCLResult +{ + VectorUnionFind unionFind; + std::vector rootSizes; + int64 nextLabel = 1; + int64 largestRoot = -1; + uint64 largestSize = 0; +}; + +template +CCLResult runForwardCCL(AbstractDataStore& store, int64 dimX, int64 dimY, int64 dimZ, ConditionFn condition, const std::atomic_bool& shouldCancel) +{ + CCLResult result; + const usize sliceSize = static_cast(dimX * dimY); + + // Rolling 2-slice buffer: only the current and previous Z-slice labels are + // kept in memory. The scanline CCL only looks at backward neighbors (x-1, + // y-1, z-1), so two slices suffice. This gives O(dimX * dimY) memory + // instead of O(volume). + std::vector labelBuffer(2 * sliceSize, 0); + // Per-label voxel count, accumulated during the forward scan so we can + // find the largest component after flattening without a separate pass. + std::vector labelSizes; + labelSizes.push_back(0); // index 0 unused (labels start at 1) + + auto sliceData = std::make_unique(sliceSize); + + for(int64 z = 0; z < dimZ; z++) + { + if(shouldCancel) + { + return result; + } + store.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(sliceData.get(), sliceSize)); + + const usize curOff = (static_cast(z) % 2) * sliceSize; + std::fill(labelBuffer.begin() + curOff, labelBuffer.begin() + curOff + sliceSize, 0); + const usize prevOff = ((static_cast(z) + 1) % 2) * sliceSize; + + for(int64 y = 0; y < dimY; y++) + { + for(int64 x = 0; x < dimX; x++) + { + const usize inSlice = static_cast(y) * static_cast(dimX) + static_cast(x); + + if(!condition(sliceData.get(), inSlice)) + { + continue; + } + + // Backward neighbor checks from label buffer + int64 nbrA = 0, nbrB = 0, nbrC = 0; + + if(x > 0) + { + nbrA = labelBuffer[curOff + inSlice - 1]; + } + if(y > 0) + { + nbrB = labelBuffer[curOff + inSlice - static_cast(dimX)]; + } + if(z > 0) + { + nbrC = labelBuffer[prevOff + inSlice]; + } + + int64 minLabel = 0; + if(nbrA > 0) + { + minLabel = nbrA; + } + if(nbrB > 0 && (minLabel == 0 || nbrB < minLabel)) + { + minLabel = nbrB; + } + if(nbrC > 0 && (minLabel == 0 || nbrC < minLabel)) + { + minLabel = nbrC; + } + + int64 assignedLabel = 0; + if(minLabel == 0) + { + assignedLabel = result.nextLabel++; + result.unionFind.makeSet(assignedLabel); + labelSizes.resize(result.nextLabel, 0); + } + else + { + assignedLabel = minLabel; + if(nbrA > 0 && nbrA != assignedLabel) + { + result.unionFind.unite(assignedLabel, nbrA); + } + if(nbrB > 0 && nbrB != assignedLabel) + { + result.unionFind.unite(assignedLabel, nbrB); + } + if(nbrC > 0 && nbrC != assignedLabel) + { + result.unionFind.unite(assignedLabel, nbrC); + } + } + + labelBuffer[curOff + inSlice] = assignedLabel; + labelSizes[assignedLabel]++; + } + } + } + + // Flatten union-find and accumulate sizes to roots + result.rootSizes.resize(result.nextLabel, 0); + for(int64 lbl = 1; lbl < result.nextLabel; lbl++) + { + int64 root = result.unionFind.find(lbl); + result.rootSizes[root] += labelSizes[lbl]; + } + + // Find largest root + for(int64 r = 1; r < result.nextLabel; r++) + { + if(result.rootSizes[r] >= result.largestSize) + { + result.largestSize = result.rootSizes[r]; + result.largestRoot = r; + } + } + + return result; +} + +// ============================================================================= +// replayForwardCCL +// ============================================================================= +// Re-derives labels by running the exact same forward CCL scan a second time +// (same Z-slice order, same scanline traversal, same union-find). Since CCL +// label assignment is fully deterministic given the same scan order and +// condition, the re-derived provisional labels match the original ones from +// runForwardCCL exactly. The union-find (already flattened) is then used to +// resolve each provisional label to its root. +// +// The `action` lambda is called for each labeled voxel with its resolved root +// label, the slice data buffer, and the voxel's (x, y, z) coordinates. It +// returns true if the slice data was modified, so the slice can be written back +// via copyFromBuffer. This allows per-voxel decisions (e.g., "mask out if +// root != largestRoot", or "fill if root is an interior hole") without ever +// storing labels for the entire volume. +// +// This is the key OOC trick: by re-computing labels on the fly using only a +// 2-slice rolling buffer, we avoid O(volume) label storage. The trade-off is +// reading the data twice, but for OOC datasets the memory savings are critical. +// +// Note: the union-find unite() calls from the first pass are not repeated here +// because the union-find is already flattened. We only need the label +// assignment logic to re-derive the same provisional labels. +// ============================================================================= +template +void replayForwardCCL(AbstractDataStore& store, int64 dimX, int64 dimY, int64 dimZ, VectorUnionFind& unionFind, ConditionFn condition, ActionFn action, const std::atomic_bool& shouldCancel) +{ + const usize sliceSize = static_cast(dimX * dimY); + auto sliceData = std::make_unique(sliceSize); + std::vector labelBuffer(2 * sliceSize, 0); + int64 nextLabel = 1; + + for(int64 z = 0; z < dimZ; z++) + { + if(shouldCancel) + { + return; + } + store.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(sliceData.get(), sliceSize)); + bool modified = false; + + const usize curOff = (static_cast(z) % 2) * sliceSize; + std::fill(labelBuffer.begin() + curOff, labelBuffer.begin() + curOff + sliceSize, 0); + const usize prevOff = ((static_cast(z) + 1) % 2) * sliceSize; + + for(int64 y = 0; y < dimY; y++) + { + for(int64 x = 0; x < dimX; x++) + { + const usize inSlice = static_cast(y) * static_cast(dimX) + static_cast(x); + + if(!condition(sliceData.get(), inSlice)) + { + continue; + } + + // Re-derive label (same logic, no union-find unites needed since already flattened) + int64 nbrA = 0, nbrB = 0, nbrC = 0; + + if(x > 0) + { + nbrA = labelBuffer[curOff + inSlice - 1]; + } + if(y > 0) + { + nbrB = labelBuffer[curOff + inSlice - static_cast(dimX)]; + } + if(z > 0) + { + nbrC = labelBuffer[prevOff + inSlice]; + } + + int64 minLabel = 0; + if(nbrA > 0) + { + minLabel = nbrA; + } + if(nbrB > 0 && (minLabel == 0 || nbrB < minLabel)) + { + minLabel = nbrB; + } + if(nbrC > 0 && (minLabel == 0 || nbrC < minLabel)) + { + minLabel = nbrC; + } + + int64 assignedLabel = 0; + if(minLabel == 0) + { + assignedLabel = nextLabel++; + } + else + { + assignedLabel = minLabel; + } + + labelBuffer[curOff + inSlice] = assignedLabel; + + // Apply the action with the re-derived label + int64 root = unionFind.find(assignedLabel); + if(action(sliceData.get(), inSlice, root, static_cast(x), static_cast(y), static_cast(z))) + { + modified = true; + } + } + } + + if(modified) + { + store.copyFromBuffer(static_cast(z) * sliceSize, nonstd::span(sliceData.get(), sliceSize)); + } + } +} + +// ============================================================================= +// IdentifySampleCCLFunctor +// ============================================================================= +// Z-slice-sequential scanline CCL implementation for identifying the largest +// connected component of good voxels in a 3D image geometry, then optionally +// filling interior holes. Processes data one Z-slice at a time using +// copyIntoBuffer/copyFromBuffer for OOC-friendly access, using a 2-slice +// rolling buffer (O(slice) memory) instead of O(volume). +// +// The algorithm has up to four phases: +// +// Phase 1: Forward CCL on good voxels +// Run runForwardCCL with condition = (goodVoxels[inSlice] == true) to +// discover all connected components and find the largest one by voxel count. +// +// Phase 2: Replay CCL to mask non-sample voxels +// Run replayForwardCCL with the same good-voxel condition. For each voxel +// whose resolved root != largestRoot, set goodVoxels to false (removing +// satellite regions/noise). No O(volume) label storage is needed -- labels +// are recomputed on the fly. +// +// Phase 3 (if fillHoles): Forward CCL on bad voxels +// Run runForwardCCL with condition = (!goodVoxels[inSlice]) to discover all +// connected components of non-sample space (potential holes + exterior). +// +// Phase 4 (if fillHoles): Replay CCL to identify and fill interior holes +// First replay: for each bad-voxel component, check if any voxel lies on +// a domain boundary. Mark boundary-touching roots in a boolean vector. +// Second replay: for each bad voxel whose root is NOT boundary-touching, +// set goodVoxels to true (filling the interior hole). +// ============================================================================= +struct IdentifySampleCCLFunctor +{ + template + void operator()(const ImageGeom* imageGeom, IDataArray* goodVoxelsPtr, bool fillHoles, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel) + { + auto& goodVoxels = goodVoxelsPtr->template getIDataStoreRefAs>(); + + SizeVec3 udims = imageGeom->getDimensions(); + const int64 dimX = static_cast(udims[0]); + const int64 dimY = static_cast(udims[1]); + const int64 dimZ = static_cast(udims[2]); + + // --- Phase 1: Forward CCL on good voxels ---------------------------------- + // Discover all connected components of good voxels and find the largest one. + // The condition lambda selects voxels where goodVoxels[inSlice] is true. + auto goodCondition = [](const T* data, usize inSlice) -> bool { return static_cast(data[inSlice]); }; + auto cclResult = runForwardCCL(goodVoxels, dimX, dimY, dimZ, goodCondition, shouldCancel); + + if(shouldCancel || cclResult.largestRoot < 0) + { + return; + } + + // --- Phase 2: Replay CCL to mask non-sample voxels ---------------------- + // Re-derive labels using a second forward pass with the same scan order + // and condition. For each voxel whose resolved root is not the largest + // component, set goodVoxels to false (removing satellite regions/noise). + // No O(volume) label storage is needed -- labels are recomputed on the fly. + const int64 largestRoot = cclResult.largestRoot; + replayForwardCCL( + goodVoxels, dimX, dimY, dimZ, cclResult.unionFind, goodCondition, + [&largestRoot](T* data, usize inSlice, int64 root, usize /*x*/, usize /*y*/, usize /*z*/) -> bool { + if(root != largestRoot) + { + data[inSlice] = static_cast(false); + return true; + } + return false; + }, + shouldCancel); + + if(shouldCancel) + { + return; + } + + // --- Phase 3: Forward CCL on bad voxels (hole detection) ----------------- + // Only runs if fillHoles is true. Discovers connected components of + // non-good voxels (the complement of the sample). These include both + // exterior empty space and interior holes. + if(fillHoles) + { + // Condition selects voxels where goodVoxels[inSlice] is false (bad data) + auto holeCondition = [](const T* data, usize inSlice) -> bool { return !static_cast(data[inSlice]); }; + auto holeCCL = runForwardCCL(goodVoxels, dimX, dimY, dimZ, holeCondition, shouldCancel); + + if(shouldCancel) + { + return; + } + + // --- Phase 4a: Replay CCL to identify boundary-touching roots --------- + // Replay the hole CCL to re-derive labels. For each labeled voxel, + // check if it lies on a domain boundary face. If so, mark its resolved + // root as boundary-touching. Components that touch the boundary are + // exterior space (not holes). This avoids O(volume) label storage by + // re-computing labels on the fly. + std::vector boundaryRoots(holeCCL.nextLabel, false); + replayForwardCCL( + goodVoxels, dimX, dimY, dimZ, holeCCL.unionFind, holeCondition, + [&boundaryRoots, dimX, dimY, dimZ](T* /*data*/, usize /*inSlice*/, int64 root, usize x, usize y, usize z) -> bool { + if(x == 0 || x == static_cast(dimX - 1) || y == 0 || y == static_cast(dimY - 1) || z == 0 || z == static_cast(dimZ - 1)) + { + boundaryRoots[root] = true; + } + return false; // Never modifies data + }, + shouldCancel); + + if(shouldCancel) + { + return; + } + + // --- Phase 4b: Replay CCL again to fill interior holes ---------------- + // A third replay of the same CCL (same condition, same union-find) to + // apply the fill. For each bad voxel whose root is NOT boundary-touching, + // it must be an interior hole fully enclosed by the sample -- set it to + // true. Boundary-touching components are exterior and left as-is. + replayForwardCCL( + goodVoxels, dimX, dimY, dimZ, holeCCL.unionFind, holeCondition, + [&boundaryRoots](T* data, usize inSlice, int64 root, usize /*x*/, usize /*y*/, usize /*z*/) -> bool { + if(!boundaryRoots[root]) + { + data[inSlice] = static_cast(true); + return true; + } + return false; + }, + shouldCancel); + } + } +}; +} // namespace + +// ----------------------------------------------------------------------------- +IdentifySampleCCL::IdentifySampleCCL(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const IdentifySampleInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +IdentifySampleCCL::~IdentifySampleCCL() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> IdentifySampleCCL::operator()() +{ + auto* inputData = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); + const auto* imageGeom = m_DataStructure.getDataAs(m_InputValues->InputImageGeometryPath); + + if(m_InputValues->SliceBySlice) + { + ExecuteDataFunction(IdentifySampleSliceBySliceFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, + static_cast(m_InputValues->SliceBySlicePlaneIndex), m_MessageHandler, m_ShouldCancel); + } + else + { + ExecuteDataFunction(IdentifySampleCCLFunctor{}, inputData->getDataType(), imageGeom, inputData, m_InputValues->FillHoles, m_MessageHandler, m_ShouldCancel); + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp new file mode 100644 index 0000000000..cba07dface --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp @@ -0,0 +1,61 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ + +struct IdentifySampleInputValues; + +/** + * @class IdentifySampleCCL + * @brief Chunk-sequential CCL algorithm for identifying the largest sample region. + * + * This is the out-of-core-optimized implementation. It uses scanline Connected + * Component Labeling (CCL) with a union-find structure, processing data in chunk + * order to minimize disk I/O. The algorithm only accesses backward neighbors + * (-X, -Y, -Z) during labeling, ensuring sequential chunk access. + * + * Trade-off: Uses a std::vector label array (8 bytes per voxel) which is + * more memory than the BFS approach (1 bit per voxel), but avoids the random + * access pattern that causes chunk thrashing in OOC mode. + * + * @see IdentifySampleBFS for the in-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + */ +class SIMPLNXCORE_EXPORT IdentifySampleCCL +{ +public: + /** + * @brief Constructs the CCL sample identification algorithm with the required context. + * @param dataStructure The data structure containing the arrays to process. + * @param mesgHandler Handler for progress and informational messages. + * @param shouldCancel Cancellation flag checked during execution. + * @param inputValues Filter parameter values controlling identification behavior. + */ + IdentifySampleCCL(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const IdentifySampleInputValues* inputValues); + ~IdentifySampleCCL() noexcept; + + IdentifySampleCCL(const IdentifySampleCCL&) = delete; + IdentifySampleCCL(IdentifySampleCCL&&) noexcept = delete; + IdentifySampleCCL& operator=(const IdentifySampleCCL&) = delete; + IdentifySampleCCL& operator=(IdentifySampleCCL&&) noexcept = delete; + + /** + * @brief Executes the CCL-based algorithm to identify the largest sample region. + * @return Result indicating success or an error with a descriptive message. + */ + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const IdentifySampleInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp new file mode 100644 index 0000000000..b6619899d3 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp @@ -0,0 +1,462 @@ +#pragma once + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" + +#include +#include +#include + +namespace nx::core +{ + +/** + * @class VectorUnionFind + * @brief Vector-based union-find for dense label sets (labels 1..N). + * + * Uses flat vectors instead of hash maps for O(1) access. Suitable for + * connected component labeling where labels are assigned sequentially. + */ +class VectorUnionFind +{ +public: + VectorUnionFind() = default; + + /** + * @brief Pre-allocates internal storage for the expected number of labels. + * @param capacity Maximum expected label value. + */ + void reserve(usize capacity) + { + m_Parent.reserve(capacity + 1); + m_Rank.reserve(capacity + 1); + } + + /** + * @brief Creates a new singleton set for label x if it does not already exist. + * @param x Label to initialize. + */ + void makeSet(int64 x) + { + if(static_cast(x) >= m_Parent.size()) + { + m_Parent.resize(x + 1, 0); + m_Rank.resize(x + 1, 0); + } + if(m_Parent[x] == 0) + { + m_Parent[x] = x; + } + } + + /** + * @brief Finds the root label with path-halving compression. + * @param x Label to find the root for. + * @return Root label of the equivalence class. + */ + int64 find(int64 x) + { + while(m_Parent[x] != x) + { + m_Parent[x] = m_Parent[m_Parent[x]]; // path halving + x = m_Parent[x]; + } + return x; + } + + /** + * @brief Merges the equivalence classes of two labels using union-by-rank. + * @param a First label. + * @param b Second label. + */ + void unite(int64 a, int64 b) + { + a = find(a); + b = find(b); + if(a == b) + { + return; + } + if(m_Rank[a] < m_Rank[b]) + { + std::swap(a, b); + } + m_Parent[b] = a; + if(m_Rank[a] == m_Rank[b]) + { + m_Rank[a]++; + } + } + +private: + std::vector m_Parent; + std::vector m_Rank; +}; + +/** + * @struct IdentifySampleSliceBySliceFunctor + * @brief BFS-based implementation for slice-by-slice mode. + * + * Slices are 2D and small relative to the full volume, so OOC chunk + * thrashing is not a concern. This functor is used by both the in-core + * and OOC algorithm classes when slice-by-slice mode is enabled. + */ +struct IdentifySampleSliceBySliceFunctor +{ + /** + * @brief Enumerates the three orthogonal slice planes. + */ + enum class Plane + { + XY, + XZ, + YZ + }; + + static constexpr int64 k_Dp1[4] = {0, 0, -1, 1}; + static constexpr int64 k_Dp2[4] = {-1, 1, 0, 0}; + + /** + * @brief BFS sample identification + optional hole filling on a single 2D slice buffer. + * + * Operates entirely on in-memory data. The sliceBuffer is modified in-place: + * non-sample voxels are set to false, and if fillHoles is true, interior + * holes (false regions not touching the boundary) are filled back to true. + */ + template + static void processSlice(T* sliceBuffer, usize sliceSize, int64 planeDim1, int64 planeDim2, bool fillHoles, const std::atomic_bool& shouldCancel) + { + // BFS for sample identification + std::vector checked(sliceSize, false); + std::vector sample(sliceSize, false); + std::vector currentVList; + int64 biggestBlock = 0; + + for(int64 p2 = 0; p2 < planeDim2; ++p2) + { + for(int64 p1 = 0; p1 < planeDim1; ++p1) + { + int64 planeIndex = p2 * planeDim1 + p1; + + if(!checked[planeIndex] && static_cast(sliceBuffer[planeIndex])) + { + currentVList.push_back(planeIndex); + int64 count = 0; + + while(count < static_cast(currentVList.size())) + { + int64 localIdx = currentVList[count]; + int64 localP1 = localIdx % planeDim1; + int64 localP2 = localIdx / planeDim1; + + for(int j = 0; j < 4; ++j) + { + int64 neighborP1 = localP1 + k_Dp1[j]; + int64 neighborP2 = localP2 + k_Dp2[j]; + + if(neighborP1 >= 0 && neighborP1 < planeDim1 && neighborP2 >= 0 && neighborP2 < planeDim2) + { + int64 neighborIdx = neighborP2 * planeDim1 + neighborP1; + + if(!checked[neighborIdx] && static_cast(sliceBuffer[neighborIdx])) + { + currentVList.push_back(neighborIdx); + checked[neighborIdx] = true; + } + } + } + count++; + } + + if(static_cast(currentVList.size()) > biggestBlock) + { + biggestBlock = currentVList.size(); + sample.assign(sliceSize, false); + for(int64 idx : currentVList) + { + sample[idx] = true; + } + } + currentVList.clear(); + } + } + } + if(shouldCancel) + { + return; + } + + // Mark non-sample voxels as false + for(usize i = 0; i < sliceSize; ++i) + { + if(!sample[i]) + { + sliceBuffer[i] = static_cast(false); + } + } + + if(shouldCancel) + { + return; + } + + // BFS for hole filling + checked.assign(sliceSize, false); + if(fillHoles) + { + for(int64 p2 = 0; p2 < planeDim2; ++p2) + { + for(int64 p1 = 0; p1 < planeDim1; ++p1) + { + int64 planeIndex = p2 * planeDim1 + p1; + + if(!checked[planeIndex] && !static_cast(sliceBuffer[planeIndex])) + { + currentVList.push_back(planeIndex); + int64 count = 0; + bool touchesBoundary = false; + + while(count < static_cast(currentVList.size())) + { + int64 localIdx = currentVList[count]; + int64 localP1 = localIdx % planeDim1; + int64 localP2 = localIdx / planeDim1; + + if(localP1 == 0 || localP1 == planeDim1 - 1 || localP2 == 0 || localP2 == planeDim2 - 1) + { + touchesBoundary = true; + } + + for(int j = 0; j < 4; ++j) + { + int64 neighborP1 = localP1 + k_Dp1[j]; + int64 neighborP2 = localP2 + k_Dp2[j]; + + if(neighborP1 >= 0 && neighborP1 < planeDim1 && neighborP2 >= 0 && neighborP2 < planeDim2) + { + int64 neighborIdx = neighborP2 * planeDim1 + neighborP1; + + if(!checked[neighborIdx] && !static_cast(sliceBuffer[neighborIdx])) + { + currentVList.push_back(neighborIdx); + checked[neighborIdx] = true; + } + } + } + count++; + } + + if(!touchesBoundary) + { + for(int64 idx : currentVList) + { + sliceBuffer[idx] = static_cast(true); + } + } + currentVList.clear(); + } + } + } + } + } + + /** + * @brief Performs BFS-based sample identification on each 2D slice of the given plane. + * @param imageGeom The image geometry providing dimensions. + * @param goodVoxelsPtr The mask array marking sample vs. non-sample voxels. + * @param fillHoles Whether to fill interior holes in each slice. + * @param plane Which orthogonal plane to slice along. + * @param messageHandler Handler for progress messages. + * @param shouldCancel Cancellation flag checked between slices. + */ + template + void operator()(const ImageGeom* imageGeom, IDataArray* goodVoxelsPtr, bool fillHoles, Plane plane, const IFilter::MessageHandler& messageHandler, const std::atomic_bool& shouldCancel) + { + auto& goodVoxels = goodVoxelsPtr->template getIDataStoreRefAs>(); + + SizeVec3 uDims = imageGeom->getDimensions(); + const int64 dimX = static_cast(uDims[0]); + const int64 dimY = static_cast(uDims[1]); + const int64 dimZ = static_cast(uDims[2]); + + int64 planeDim1 = 0, planeDim2 = 0, fixedDim = 0; + int64 stride1 = 0, stride2 = 0, fixedStride = 0; + + switch(plane) + { + case Plane::XY: + planeDim1 = dimX; + planeDim2 = dimY; + fixedDim = dimZ; + stride1 = 1; + stride2 = dimX; + fixedStride = dimX * dimY; + break; + + case Plane::XZ: + planeDim1 = dimX; + planeDim2 = dimZ; + fixedDim = dimY; + stride1 = 1; + stride2 = dimX * dimY; + fixedStride = dimX; + break; + + case Plane::YZ: + planeDim1 = dimY; + planeDim2 = dimZ; + fixedDim = dimX; + stride1 = dimX; + stride2 = dimX * dimY; + fixedStride = 1; + break; + } + + const usize sliceSize = static_cast(planeDim1 * planeDim2); + + // YZ batched path: read each Z-slice once per batch instead of once per X + // position. This reduces HDF5 ops from fixedDim × dimZ to + // ceil(fixedDim/batch) × dimZ × 3, giving ~10x speedup for OOC. + if constexpr(!std::is_same_v) + { + if(plane == Plane::YZ) + { + constexpr int64 k_BatchSize = 64; + const usize zSliceElements = static_cast(dimX * dimY); + std::vector zSliceBuf(zSliceElements); + + for(int64 batchStart = 0; batchStart < fixedDim; batchStart += k_BatchSize) + { + if(shouldCancel) + { + return; + } + + const int64 batchEnd = std::min(batchStart + k_BatchSize, fixedDim); + const int64 batchCount = batchEnd - batchStart; + + // Allocate column buffers for this batch + std::vector> columnBuffers(static_cast(batchCount)); + for(int64 b = 0; b < batchCount; b++) + { + columnBuffers[static_cast(b)] = std::make_unique(sliceSize); + } + + // Phase A: Read each Z-slice once, extract columns for all batch members + for(int64 z = 0; z < dimZ; z++) + { + goodVoxels.copyIntoBuffer(static_cast(z) * zSliceElements, nonstd::span(zSliceBuf.data(), zSliceElements)); + for(int64 b = 0; b < batchCount; b++) + { + const int64 x = batchStart + b; + for(int64 y = 0; y < dimY; y++) + { + columnBuffers[static_cast(b)][static_cast(z * dimY + y)] = zSliceBuf[static_cast(y * dimX + x)]; + } + } + } + + // Phase B: BFS each column buffer independently + for(int64 b = 0; b < batchCount; b++) + { + if(shouldCancel) + { + return; + } + messageHandler(IFilter::Message::Type::Info, fmt::format("Slice {}", batchStart + b)); + processSlice(columnBuffers[static_cast(b)].get(), sliceSize, planeDim1, planeDim2, fillHoles, shouldCancel); + } + + // Phase C: Write back — read each Z-slice, insert columns, write + for(int64 z = 0; z < dimZ; z++) + { + goodVoxels.copyIntoBuffer(static_cast(z) * zSliceElements, nonstd::span(zSliceBuf.data(), zSliceElements)); + for(int64 b = 0; b < batchCount; b++) + { + const int64 x = batchStart + b; + for(int64 y = 0; y < dimY; y++) + { + zSliceBuf[static_cast(y * dimX + x)] = columnBuffers[static_cast(b)][static_cast(z * dimY + y)]; + } + } + goodVoxels.copyFromBuffer(static_cast(z) * zSliceElements, nonstd::span(zSliceBuf.data(), zSliceElements)); + } + } + return; // YZ batched processing complete + } + } + + // XY / XZ / YZ-bool path: process one plane at a time + auto sliceBuffer = std::make_unique(sliceSize); + + for(int64 fixedIdx = 0; fixedIdx < fixedDim; ++fixedIdx) + { + if(shouldCancel) + { + return; + } + messageHandler(IFilter::Message::Type::Info, fmt::format("Slice {}", fixedIdx)); + + // Read the 2D slice into a local buffer using bulk reads where possible. + if(stride1 == 1 && stride2 == planeDim1) + { + // XY plane: entire slice is contiguous in memory. Single bulk read. + goodVoxels.copyIntoBuffer(static_cast(fixedIdx * fixedStride), nonstd::span(sliceBuffer.get(), sliceSize)); + } + else if(stride1 == 1) + { + // XZ plane: each row of planeDim1 elements is contiguous, but rows + // are separated by stride2 (dimX*dimY). Read row-by-row. + for(int64 p2 = 0; p2 < planeDim2; ++p2) + { + usize rowStart = static_cast(fixedIdx * fixedStride + p2 * stride2); + goodVoxels.copyIntoBuffer(rowStart, nonstd::span(sliceBuffer.get() + p2 * planeDim1, static_cast(planeDim1))); + } + } + else + { + // YZ plane (bool type fallback): per-element access. + for(int64 p2 = 0; p2 < planeDim2; ++p2) + { + for(int64 p1 = 0; p1 < planeDim1; ++p1) + { + sliceBuffer[static_cast(p2 * planeDim1 + p1)] = goodVoxels.getValue(static_cast(fixedIdx * fixedStride + p2 * stride2 + p1 * stride1)); + } + } + } + + processSlice(sliceBuffer.get(), sliceSize, planeDim1, planeDim2, fillHoles, shouldCancel); + + // Write the modified slice back to the DataStore using bulk writes where possible. + if(stride1 == 1 && stride2 == planeDim1) + { + // XY plane: entire slice is contiguous in memory. Single bulk write. + goodVoxels.copyFromBuffer(static_cast(fixedIdx * fixedStride), nonstd::span(sliceBuffer.get(), sliceSize)); + } + else if(stride1 == 1) + { + // XZ plane: each row of planeDim1 elements is contiguous. Write row-by-row. + for(int64 p2 = 0; p2 < planeDim2; ++p2) + { + usize rowStart = static_cast(fixedIdx * fixedStride + p2 * stride2); + goodVoxels.copyFromBuffer(rowStart, nonstd::span(sliceBuffer.get() + p2 * planeDim1, static_cast(planeDim1))); + } + } + else + { + // YZ plane (bool type fallback): per-element write-back. + for(int64 p2 = 0; p2 < planeDim2; ++p2) + { + for(int64 p1 = 0; p1 < planeDim1; ++p1) + { + goodVoxels.setValue(static_cast(fixedIdx * fixedStride + p2 * stride2 + p1 * stride1), sliceBuffer[static_cast(p2 * planeDim1 + p1)]); + } + } + } + } + } +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp new file mode 100644 index 0000000000..e50e7e395b --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp @@ -0,0 +1,35 @@ +#include "MultiThresholdObjects.hpp" + +#include "MultiThresholdObjectsDirect.hpp" +#include "MultiThresholdObjectsScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +MultiThresholdObjects::MultiThresholdObjects(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + MultiThresholdObjectsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +MultiThresholdObjects::~MultiThresholdObjects() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> MultiThresholdObjects::operator()() +{ + auto thresholdsObject = m_InputValues->ArrayThresholdsObject; + const auto& requiredPaths = thresholdsObject.getRequiredPaths(); + const IDataArray* checkArray = nullptr; + if(!requiredPaths.empty()) + { + checkArray = m_DataStructure.getDataAs(*requiredPaths.begin()); + } + return DispatchAlgorithm({checkArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp new file mode 100644 index 0000000000..2371efc102 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp @@ -0,0 +1,67 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/ArrayThresholdsParameter.hpp" +#include "simplnx/Parameters/BoolParameter.hpp" +#include "simplnx/Parameters/DataObjectNameParameter.hpp" +#include "simplnx/Parameters/DataTypeParameter.hpp" +#include "simplnx/Parameters/NumberParameter.hpp" + +/** +* This is example code to put in the Execute Method of the filter. + MultiThresholdObjectsInputValues inputValues; + inputValues.ArrayThresholdsObject = filterArgs.value(array_thresholds_object); + inputValues.CreatedMaskType = filterArgs.value(created_mask_type); + inputValues.CustomFalseValue = filterArgs.value(custom_false_value); + inputValues.CustomTrueValue = filterArgs.value(custom_true_value); + inputValues.OutputDataArrayName = filterArgs.value(output_data_array_name); + inputValues.UseCustomFalseValue = filterArgs.value(use_custom_false_value); + inputValues.UseCustomTrueValue = filterArgs.value(use_custom_true_value); + return MultiThresholdObjects(dataStructure, messageHandler, shouldCancel, &inputValues)(); + +*/ + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT MultiThresholdObjectsInputValues +{ + ArrayThresholdsParameter::ValueType ArrayThresholdsObject; + DataTypeParameter::ValueType CreatedMaskType; + Float64Parameter::ValueType CustomFalseValue; + Float64Parameter::ValueType CustomTrueValue; + DataObjectNameParameter::ValueType OutputDataArrayName; + BoolParameter::ValueType UseCustomFalseValue; + BoolParameter::ValueType UseCustomTrueValue; +}; + +/** + * @class MultiThresholdObjects + * @brief This algorithm implements support code for the MultiThresholdObjectsFilter + */ + +class SIMPLNXCORE_EXPORT MultiThresholdObjects +{ +public: + MultiThresholdObjects(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues); + ~MultiThresholdObjects() noexcept; + + MultiThresholdObjects(const MultiThresholdObjects&) = delete; + MultiThresholdObjects(MultiThresholdObjects&&) noexcept = delete; + MultiThresholdObjects& operator=(const MultiThresholdObjects&) = delete; + MultiThresholdObjects& operator=(MultiThresholdObjects&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const MultiThresholdObjectsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp index fcf95d0b0d..3ba076ba8c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp @@ -1,5 +1,7 @@ #include "MultiThresholdObjectsDirect.hpp" +#include "MultiThresholdObjects.hpp" + #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/Utilities/ArrayThreshold.hpp" @@ -26,9 +28,9 @@ class ThresholdFilterHelper template void filterDataWithComparision(const AbstractDataStore& m_Input, T trueValue, T falseValue) { - size_t numTuples = m_Input.getNumberOfTuples(); + usize numTuples = m_Input.getNumberOfTuples(); T value = static_cast(m_ComparisonValue); - for(size_t tupleIndex = 0; tupleIndex < numTuples; ++tupleIndex) + for(usize tupleIndex = 0; tupleIndex < numTuples; ++tupleIndex) { T inputValue = m_Input.getComponentValue(tupleIndex, m_ComponentIndex); T outputValue = CompT{}(inputValue, value) ? trueValue : falseValue; @@ -110,11 +112,11 @@ void InsertThreshold(usize numItems, AbstractDataStore& currentStore, nx::cor } template -void ThresholdValue(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32_t& err, bool replaceInput, bool inverse, T trueValue, - T falseValue) +void ThresholdValue(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32& err, bool replaceInput, bool inverse, T trueValue, + T falseValue, const std::atomic_bool& shouldCancel) { // Get the total number of tuples, create and initialize an array with FALSE to use for these results - size_t totalTuples = outputResultStore.getNumberOfTuples(); + usize totalTuples = outputResultStore.getNumberOfTuples(); std::vector tempResultVector(totalTuples, falseValue); nx::core::ArrayThreshold::ComparisonType compOperator = comparisonValue.getComparisonType(); @@ -131,6 +133,11 @@ void ThresholdValue(const ArrayThreshold& comparisonValue, const DataStructure& ExecuteDataFunction(ExecuteThresholdHelper{}, iDataArray.getDataType(), helper, iDataArray, trueValue, falseValue); + if(shouldCancel) + { + return; + } + if(replaceInput) { if(inverse) @@ -138,7 +145,7 @@ void ThresholdValue(const ArrayThreshold& comparisonValue, const DataStructure& std::reverse(tempResultVector.begin(), tempResultVector.end()); } // copy the temp uint8 vector to the final uint8 result array - for(size_t i = 0; i < totalTuples; i++) + for(usize i = 0; i < totalTuples; i++) { outputResultStore[i] = tempResultVector[i]; } @@ -153,20 +160,19 @@ void ThresholdValue(const ArrayThreshold& comparisonValue, const DataStructure& struct ThresholdValueFunctor { template - void operator()(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, IDataArray& outputResultArray, int32_t& err, bool replaceInput, bool inverse, T trueValue, T falseValue) + void operator()(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, IDataArray& outputResultArray, int32& err, bool replaceInput, bool inverse, T trueValue, T falseValue, + const std::atomic_bool& shouldCancel) { - // Traditionally we would do a check to ensure we get a valid pointer, I'm forgoing that check because it - // was essentially done in the preflight part. - ThresholdValue(comparisonValue, dataStructure, outputResultArray.template getIDataStoreRefAs>(), err, replaceInput, inverse, trueValue, falseValue); + ThresholdValue(comparisonValue, dataStructure, outputResultArray.template getIDataStoreRefAs>(), err, replaceInput, inverse, trueValue, falseValue, shouldCancel); } }; template -void ThresholdSet(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32_t& err, bool replaceInput, bool inverse, T trueValue, - T falseValue) +void ThresholdSet(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32& err, bool replaceInput, bool inverse, T trueValue, + T falseValue, const std::atomic_bool& shouldCancel) { // Get the total number of tuples, create and initialize an array with FALSE to use for these results - size_t totalTuples = outputResultStore.getNumberOfTuples(); + usize totalTuples = outputResultStore.getNumberOfTuples(); std::vector tempResultVector(totalTuples, falseValue); bool firstValueFound = false; @@ -174,19 +180,29 @@ void ThresholdSet(const ArrayThresholdSet& inputComparisonSet, const DataStructu ArrayThresholdSet::CollectionType thresholds = inputComparisonSet.getArrayThresholds(); for(const std::shared_ptr& threshold : thresholds) { + if(shouldCancel) + { + return; + } + const IArrayThreshold* thresholdPtr = threshold.get(); if(const auto* comparisonSet = dynamic_cast(thresholdPtr); comparisonSet != nullptr) { - ThresholdSet(*comparisonSet, dataStructure, outputResultStore, err, !firstValueFound, false, trueValue, falseValue); + ThresholdSet(*comparisonSet, dataStructure, outputResultStore, err, !firstValueFound, false, trueValue, falseValue, shouldCancel); firstValueFound = true; } else if(const auto* comparisonValue = dynamic_cast(thresholdPtr); comparisonValue != nullptr) { - ThresholdValue(*comparisonValue, dataStructure, outputResultStore, err, !firstValueFound, false, trueValue, falseValue); + ThresholdValue(*comparisonValue, dataStructure, outputResultStore, err, !firstValueFound, false, trueValue, falseValue, shouldCancel); firstValueFound = true; } } + if(shouldCancel) + { + return; + } + if(replaceInput) { if(inverse) @@ -194,7 +210,7 @@ void ThresholdSet(const ArrayThresholdSet& inputComparisonSet, const DataStructu std::reverse(tempResultVector.begin(), tempResultVector.end()); } // copy the temp uint8 vector to the final uint8 result array - for(size_t i = 0; i < totalTuples; i++) + for(usize i = 0; i < totalTuples; i++) { outputResultStore[i] = tempResultVector[i]; } @@ -209,19 +225,17 @@ void ThresholdSet(const ArrayThresholdSet& inputComparisonSet, const DataStructu struct ThresholdSetFunctor { template - void operator()(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, IDataArray& outputResultArray, int32_t& err, bool replaceInput, bool inverse, T trueValue, - T falseValue) + void operator()(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, IDataArray& outputResultArray, int32& err, bool replaceInput, bool inverse, T trueValue, + T falseValue, const std::atomic_bool& shouldCancel) { - // Traditionally we would do a check to ensure we get a valid pointer, I'm forgoing that check because it - // was essentially done in the preflight part. - ThresholdSet(inputComparisonSet, dataStructure, outputResultArray.template getIDataStoreRefAs>(), err, replaceInput, inverse, trueValue, falseValue); + ThresholdSet(inputComparisonSet, dataStructure, outputResultArray.template getIDataStoreRefAs>(), err, replaceInput, inverse, trueValue, falseValue, shouldCancel); } }; } // namespace // ----------------------------------------------------------------------------- MultiThresholdObjectsDirect::MultiThresholdObjectsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, - MultiThresholdObjectsInputValues* inputValues) + const MultiThresholdObjectsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -246,27 +260,36 @@ Result<> MultiThresholdObjectsDirect::operator()() float64 trueValue = useCustomTrueValue ? customTrueValue : 1.0; float64 falseValue = useCustomFalseValue ? customFalseValue : 0.0; - bool firstValueFound = false; DataPath maskArrayPath = (*thresholdsObject.getRequiredPaths().begin()).replaceName(maskArrayName); - int32_t err = 0; - ArrayThresholdSet::CollectionType thresholdSet = thresholdsObject.getArrayThresholds(); - for(const std::shared_ptr& threshold : thresholdSet) + ArrayThresholdSet::CollectionType ThresholdSet = thresholdsObject.getArrayThresholds(); + + usize numThresholds = ThresholdSet.size(); + const auto& maskArray = m_DataStructure.getDataRefAs(maskArrayPath); + usize totalTuples = maskArray.getNumberOfTuples(); + + m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Applying {} threshold{} to {} tuples...", numThresholds, numThresholds == 1 ? "" : "s", totalTuples)); + + bool firstValueFound = false; + int32 err = 0; + for(usize threshIdx = 0; threshIdx < ThresholdSet.size(); threshIdx++) { if(m_ShouldCancel) { return {}; } + + const auto& threshold = ThresholdSet[threshIdx]; const IArrayThreshold* thresholdPtr = threshold.get(); if(const auto* comparisonSet = dynamic_cast(thresholdPtr); comparisonSet != nullptr) { ExecuteDataFunction(ThresholdSetFunctor{}, maskArrayType, *comparisonSet, m_DataStructure, m_DataStructure.getDataRefAs(maskArrayPath), err, !firstValueFound, - thresholdsObject.isInverted(), trueValue, falseValue); + thresholdsObject.isInverted(), trueValue, falseValue, m_ShouldCancel); firstValueFound = true; } else if(const auto* comparisonValue = dynamic_cast(thresholdPtr); comparisonValue != nullptr) { ExecuteDataFunction(ThresholdValueFunctor{}, maskArrayType, *comparisonValue, m_DataStructure, m_DataStructure.getDataRefAs(maskArrayPath), err, !firstValueFound, - thresholdsObject.isInverted(), trueValue, falseValue); + thresholdsObject.isInverted(), trueValue, falseValue, m_ShouldCancel); firstValueFound = true; } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp index eaef405c9f..ac17007ac8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp @@ -2,52 +2,23 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/ArrayThresholdsParameter.hpp" -#include "simplnx/Parameters/BoolParameter.hpp" -#include "simplnx/Parameters/DataObjectNameParameter.hpp" -#include "simplnx/Parameters/DataTypeParameter.hpp" -#include "simplnx/Parameters/NumberParameter.hpp" - -/** -* This is example code to put in the Execute Method of the filter. - MultiThresholdObjectsInputValues inputValues; - inputValues.ArrayThresholdsObject = filterArgs.value(array_thresholds_object); - inputValues.CreatedMaskType = filterArgs.value(created_mask_type); - inputValues.CustomFalseValue = filterArgs.value(custom_false_value); - inputValues.CustomTrueValue = filterArgs.value(custom_true_value); - inputValues.OutputDataArrayName = filterArgs.value(output_data_array_name); - inputValues.UseCustomFalseValue = filterArgs.value(use_custom_false_value); - inputValues.UseCustomTrueValue = filterArgs.value(use_custom_true_value); - return MultiThresholdObjectsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); - -*/ namespace nx::core { - -struct SIMPLNXCORE_EXPORT MultiThresholdObjectsInputValues -{ - ArrayThresholdsParameter::ValueType ArrayThresholdsObject; - DataTypeParameter::ValueType CreatedMaskType; - Float64Parameter::ValueType CustomFalseValue; - Float64Parameter::ValueType CustomTrueValue; - DataObjectNameParameter::ValueType OutputDataArrayName; - BoolParameter::ValueType UseCustomFalseValue; - BoolParameter::ValueType UseCustomTrueValue; -}; +struct MultiThresholdObjectsInputValues; /** * @class MultiThresholdObjectsDirect - * @brief This algorithm implements support code for the MultiThresholdObjectsFilter + * @brief In-core algorithm for MultiThresholdObjects. Preserves the original per-element + * access pattern and O(n) tempResultVector allocation. Selected by DispatchAlgorithm + * when all input arrays are backed by in-memory DataStore. */ - class SIMPLNXCORE_EXPORT MultiThresholdObjectsDirect { public: - MultiThresholdObjectsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues); + MultiThresholdObjectsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const MultiThresholdObjectsInputValues* inputValues); ~MultiThresholdObjectsDirect() noexcept; MultiThresholdObjectsDirect(const MultiThresholdObjectsDirect&) = delete; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp new file mode 100644 index 0000000000..ee6da40666 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp @@ -0,0 +1,285 @@ +#include "MultiThresholdObjectsScanline.hpp" + +#include "MultiThresholdObjects.hpp" + +#include "simplnx/Common/TypeTraits.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/ArrayThreshold.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" + +#include + +#include +#include + +using namespace nx::core; + +namespace +{ +constexpr usize k_ChunkSize = 65536; + +/** + * @brief Applies a single comparison operator to a chunk of input data, writing + * trueValue/falseValue into the chunk-sized output buffer. + * + * @tparam CompT Comparison functor (std::less<>, std::greater<>, etc.) + * @tparam InputT The input array element type + * @tparam MaskT The output mask element type + */ +template +void filterChunkWithComparison(const InputT* inputBuffer, usize numComponents, usize componentIndex, usize chunkTuples, MaskT trueValue, MaskT falseValue, MaskT* outputBuffer, InputT comparisonValue) +{ + for(usize i = 0; i < chunkTuples; ++i) + { + InputT inputValue = inputBuffer[i * numComponents + componentIndex]; + outputBuffer[i] = CompT{}(inputValue, comparisonValue) ? trueValue : falseValue; + } +} + +/** + * @brief Dispatches the comparison based on the ComparisonType enum. + */ +template +void filterChunk(ArrayThreshold::ComparisonType compOperator, const InputT* inputBuffer, usize numComponents, usize componentIndex, usize chunkTuples, MaskT trueValue, MaskT falseValue, + MaskT* outputBuffer, InputT comparisonValue) +{ + switch(compOperator) + { + case ArrayThreshold::ComparisonType::LessThan: + filterChunkWithComparison, InputT, MaskT>(inputBuffer, numComponents, componentIndex, chunkTuples, trueValue, falseValue, outputBuffer, comparisonValue); + break; + case ArrayThreshold::ComparisonType::GreaterThan: + filterChunkWithComparison, InputT, MaskT>(inputBuffer, numComponents, componentIndex, chunkTuples, trueValue, falseValue, outputBuffer, comparisonValue); + break; + case ArrayThreshold::ComparisonType::Operator_Equal: + filterChunkWithComparison, InputT, MaskT>(inputBuffer, numComponents, componentIndex, chunkTuples, trueValue, falseValue, outputBuffer, comparisonValue); + break; + case ArrayThreshold::ComparisonType::Operator_NotEqual: + filterChunkWithComparison, InputT, MaskT>(inputBuffer, numComponents, componentIndex, chunkTuples, trueValue, falseValue, outputBuffer, comparisonValue); + break; + default: { + std::string errorMessage = fmt::format("MultiThresholdObjects Comparison Operator not understood: '{}'", static_cast(compOperator)); + throw std::runtime_error(errorMessage); + } + } +} + +/** + * @brief Merges a chunk of new threshold results into the current output chunk. + */ +template +void insertThresholdChunk(usize chunkTuples, MaskT* currentBuffer, IArrayThreshold::UnionOperator unionOperator, MaskT* newBuffer, bool inverse, MaskT trueValue, MaskT falseValue) +{ + for(usize i = 0; i < chunkTuples; i++) + { + if(inverse) + { + newBuffer[i] = (newBuffer[i] == trueValue) ? falseValue : trueValue; + } + + if(IArrayThreshold::UnionOperator::Or == unionOperator) + { + currentBuffer[i] = (currentBuffer[i] == trueValue || newBuffer[i] == trueValue) ? trueValue : falseValue; + } + else if(currentBuffer[i] == falseValue || newBuffer[i] == falseValue) + { + currentBuffer[i] = falseValue; + } + } +} + +/** + * @brief Functor that reads a chunk of the input array via copyIntoBuffer and + * applies the threshold comparison to produce chunk-sized output. + */ +struct ChunkedThresholdHelper +{ + template + void operator()(const IDataArray& iDataArray, ArrayThreshold::ComparisonType compOperator, ArrayThreshold::ComparisonValue compValue, usize componentIndex, usize chunkStartTuple, usize chunkTuples, + MaskT trueValue, MaskT falseValue, MaskT* tempBuffer) + { + const auto& inputStore = iDataArray.template getIDataStoreRefAs>(); + usize numComponents = inputStore.getNumberOfComponents(); + + // Read input chunk (flat elements = tuples * components) + // Use unique_ptr instead of vector to avoid std::vector specialization + usize flatStart = chunkStartTuple * numComponents; + usize flatCount = chunkTuples * numComponents; + auto inputBuffer = std::make_unique(flatCount); + inputStore.copyIntoBuffer(flatStart, nonstd::span(inputBuffer.get(), flatCount)); + + InputT comparisonValueTyped = static_cast(compValue); + filterChunk(compOperator, inputBuffer.get(), numComponents, componentIndex, chunkTuples, trueValue, falseValue, tempBuffer, comparisonValueTyped); + } +}; + +/** + * @brief Processes a single ArrayThreshold in chunks for OOC. + */ +template +void ThresholdValueChunked(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32& err, bool replaceInput, bool inverse, + MaskT trueValue, MaskT falseValue, const std::atomic_bool& shouldCancel) +{ + usize totalTuples = outputResultStore.getNumberOfTuples(); + + ArrayThreshold::ComparisonType compOperator = comparisonValue.getComparisonType(); + ArrayThreshold::ComparisonValue compValue = comparisonValue.getComparisonValue(); + IArrayThreshold::UnionOperator unionOperator = comparisonValue.getUnionOperator(); + + DataPath inputDataArrayPath = comparisonValue.getArrayPath(); + usize componentIndex = comparisonValue.getComponentIndex(); + + const auto& iDataArray = dataStructure.getDataRefAs(inputDataArrayPath); + DataType inputDataType = iDataArray.getDataType(); + + // Process in chunks + for(usize chunkStart = 0; chunkStart < totalTuples; chunkStart += k_ChunkSize) + { + if(shouldCancel) + { + return; + } + + usize chunkTuples = std::min(k_ChunkSize, totalTuples - chunkStart); + + // Chunk-sized temp buffer for this threshold's results + // Use unique_ptr instead of vector to avoid std::vector specialization + auto tempBuffer = std::make_unique(chunkTuples); + std::fill_n(tempBuffer.get(), chunkTuples, falseValue); + + // Apply threshold comparison to this chunk + ExecuteDataFunction(ChunkedThresholdHelper{}, inputDataType, iDataArray, compOperator, compValue, componentIndex, chunkStart, chunkTuples, trueValue, falseValue, tempBuffer.get()); + + if(replaceInput) + { + if(inverse) + { + for(usize i = 0; i < chunkTuples; i++) + { + tempBuffer[i] = (tempBuffer[i] == trueValue) ? falseValue : trueValue; + } + } + // Write temp buffer directly to output store + outputResultStore.copyFromBuffer(chunkStart, nonstd::span(tempBuffer.get(), chunkTuples)); + } + else + { + // Read current output chunk, merge, write back + auto currentBuffer = std::make_unique(chunkTuples); + outputResultStore.copyIntoBuffer(chunkStart, nonstd::span(currentBuffer.get(), chunkTuples)); + insertThresholdChunk(chunkTuples, currentBuffer.get(), unionOperator, tempBuffer.get(), inverse, trueValue, falseValue); + outputResultStore.copyFromBuffer(chunkStart, nonstd::span(currentBuffer.get(), chunkTuples)); + } + } +} + +struct ThresholdValueChunkedFunctor +{ + template + void operator()(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, IDataArray& outputResultArray, int32& err, bool replaceInput, bool inverse, MaskT trueValue, + MaskT falseValue, const std::atomic_bool& shouldCancel) + { + ThresholdValueChunked(comparisonValue, dataStructure, outputResultArray.template getIDataStoreRefAs>(), err, replaceInput, inverse, trueValue, falseValue, shouldCancel); + } +}; + +/** + * @brief Processes an ArrayThresholdSet in chunks for OOC. + */ +template +void ThresholdSetChunked(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32& err, bool replaceInput, bool inverse, + MaskT trueValue, MaskT falseValue, const std::atomic_bool& shouldCancel) +{ + bool firstValueFound = false; + + ArrayThresholdSet::CollectionType thresholds = inputComparisonSet.getArrayThresholds(); + for(const std::shared_ptr& threshold : thresholds) + { + if(shouldCancel) + { + return; + } + + const IArrayThreshold* thresholdPtr = threshold.get(); + if(const auto* comparisonSet = dynamic_cast(thresholdPtr); comparisonSet != nullptr) + { + ThresholdSetChunked(*comparisonSet, dataStructure, outputResultStore, err, !firstValueFound, false, trueValue, falseValue, shouldCancel); + firstValueFound = true; + } + else if(const auto* comparisonValue = dynamic_cast(thresholdPtr); comparisonValue != nullptr) + { + ThresholdValueChunked(*comparisonValue, dataStructure, outputResultStore, err, !firstValueFound, false, trueValue, falseValue, shouldCancel); + firstValueFound = true; + } + } +} + +struct ThresholdSetChunkedFunctor +{ + template + void operator()(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, IDataArray& outputResultArray, int32& err, bool replaceInput, bool inverse, MaskT trueValue, + MaskT falseValue, const std::atomic_bool& shouldCancel) + { + ThresholdSetChunked(inputComparisonSet, dataStructure, outputResultArray.template getIDataStoreRefAs>(), err, replaceInput, inverse, trueValue, falseValue, + shouldCancel); + } +}; +} // namespace + +// ----------------------------------------------------------------------------- +MultiThresholdObjectsScanline::MultiThresholdObjectsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const MultiThresholdObjectsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +MultiThresholdObjectsScanline::~MultiThresholdObjectsScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> MultiThresholdObjectsScanline::operator()() +{ + auto thresholdsObject = m_InputValues->ArrayThresholdsObject; + auto maskArrayName = m_InputValues->OutputDataArrayName; + auto maskArrayType = m_InputValues->CreatedMaskType; + auto useCustomTrueValue = m_InputValues->UseCustomTrueValue; + auto useCustomFalseValue = m_InputValues->UseCustomFalseValue; + auto customTrueValue = m_InputValues->CustomTrueValue; + auto customFalseValue = m_InputValues->CustomFalseValue; + + float64 trueValue = useCustomTrueValue ? customTrueValue : 1.0; + float64 falseValue = useCustomFalseValue ? customFalseValue : 0.0; + + DataPath maskArrayPath = (*thresholdsObject.getRequiredPaths().begin()).replaceName(maskArrayName); + ArrayThresholdSet::CollectionType ThresholdSet = thresholdsObject.getArrayThresholds(); + + bool firstValueFound = false; + int32 err = 0; + for(usize threshIdx = 0; threshIdx < ThresholdSet.size(); threshIdx++) + { + if(m_ShouldCancel) + { + return {}; + } + + const auto& threshold = ThresholdSet[threshIdx]; + const IArrayThreshold* thresholdPtr = threshold.get(); + if(const auto* comparisonSet = dynamic_cast(thresholdPtr); comparisonSet != nullptr) + { + ExecuteDataFunction(ThresholdSetChunkedFunctor{}, maskArrayType, *comparisonSet, m_DataStructure, m_DataStructure.getDataRefAs(maskArrayPath), err, !firstValueFound, + thresholdsObject.isInverted(), trueValue, falseValue, m_ShouldCancel); + firstValueFound = true; + } + else if(const auto* comparisonValue = dynamic_cast(thresholdPtr); comparisonValue != nullptr) + { + ExecuteDataFunction(ThresholdValueChunkedFunctor{}, maskArrayType, *comparisonValue, m_DataStructure, m_DataStructure.getDataRefAs(maskArrayPath), err, !firstValueFound, + thresholdsObject.isInverted(), trueValue, falseValue, m_ShouldCancel); + firstValueFound = true; + } + } + + return {}; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp new file mode 100644 index 0000000000..7f6837ae10 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp @@ -0,0 +1,39 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +namespace nx::core +{ +struct MultiThresholdObjectsInputValues; + +/** + * @class MultiThresholdObjectsScanline + * @brief Out-of-core algorithm for MultiThresholdObjects. Processes data in fixed-size + * chunks using copyIntoBuffer/copyFromBuffer bulk I/O to avoid per-element OOC access + * and eliminates the O(n) tempResultVector allocation. Selected by DispatchAlgorithm + * when any input array is backed by ZarrStore (out-of-core storage). + */ +class SIMPLNXCORE_EXPORT MultiThresholdObjectsScanline +{ +public: + MultiThresholdObjectsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const MultiThresholdObjectsInputValues* inputValues); + ~MultiThresholdObjectsScanline() noexcept; + + MultiThresholdObjectsScanline(const MultiThresholdObjectsScanline&) = delete; + MultiThresholdObjectsScanline(MultiThresholdObjectsScanline&&) noexcept = delete; + MultiThresholdObjectsScanline& operator=(const MultiThresholdObjectsScanline&) = delete; + MultiThresholdObjectsScanline& operator=(MultiThresholdObjectsScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const MultiThresholdObjectsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp new file mode 100644 index 0000000000..fd79f774c1 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp @@ -0,0 +1,27 @@ +#include "QuickSurfaceMesh.hpp" +#include "QuickSurfaceMeshDirect.hpp" +#include "QuickSurfaceMeshScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +QuickSurfaceMesh::QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +QuickSurfaceMesh::~QuickSurfaceMesh() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> QuickSurfaceMesh::operator()() +{ + auto* featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + return DispatchAlgorithm({featureIds}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp new file mode 100644 index 0000000000..a906eb20ba --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp @@ -0,0 +1,62 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/MultiArraySelectionParameter.hpp" + +#include + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT QuickSurfaceMeshInputValues +{ + bool FixProblemVoxels; + bool RepairTriangleWinding; + bool GenerateTripleLines; + + DataPath GridGeomDataPath; + DataPath FeatureIdsArrayPath; + MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; + MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; + DataPath TriangleGeometryPath; + DataPath VertexGroupDataPath; + DataPath NodeTypesDataPath; + DataPath FaceGroupDataPath; + DataPath FaceLabelsDataPath; + MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; +}; + +/** + * @class QuickSurfaceMesh + * @brief Dispatcher that selects between QuickSurfaceMeshDirect (in-core) and + * QuickSurfaceMeshScanline (OOC) based on the storage type of input arrays. + */ +class SIMPLNXCORE_EXPORT QuickSurfaceMesh +{ +public: + using VertexStore = AbstractDataStore; + using TriStore = AbstractDataStore; + using MeshIndexType = IGeometry::MeshIndexType; + + QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler); + ~QuickSurfaceMesh() noexcept; + + QuickSurfaceMesh(const QuickSurfaceMesh&) = delete; + QuickSurfaceMesh(QuickSurfaceMesh&&) noexcept = delete; + QuickSurfaceMesh& operator=(const QuickSurfaceMesh&) = delete; + QuickSurfaceMesh& operator=(QuickSurfaceMesh&&) noexcept = delete; + + Result<> operator()(); + +private: + DataStructure& m_DataStructure; + const QuickSurfaceMeshInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp index 3efec65f45..a3bc17a521 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp @@ -1,4 +1,6 @@ #include "QuickSurfaceMeshDirect.hpp" + +#include "QuickSurfaceMesh.hpp" #include "TupleTransfer.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -7,225 +9,24 @@ #include "simplnx/DataStructure/Geometry/TriangleGeom.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" #include "simplnx/Utilities/Meshing/TriangleUtilities.hpp" -#include "simplnx/Utilities/ParallelData3DAlgorithm.hpp" -#include -#include +#include +#include +#include using namespace nx::core; // ----------------------------------------------------------------------------- namespace { -constexpr double k_RangeMin = 0.0; -constexpr double k_RangeMax = 1.0; -std::random_device k_RandomDevice; // Will be used to obtain a seed for the random number engine -std::mt19937_64 k_Generator(k_RandomDevice()); // Standard mersenne_twister_engine seeded with rd() -std::mt19937_64::result_type k_Seed = 3412341234123412; -std::uniform_real_distribution<> k_Distribution(k_RangeMin, k_RangeMax); - -template -void hashCombine(size_t& seed, const T& obj) -{ - std::hash hasher; - seed ^= hasher(obj) + 0x9e3779b9 + (seed << 6) + (seed >> 2); -} +constexpr float64 k_RangeMin = 0.0; +constexpr float64 k_RangeMax = 1.0; +constexpr std::mt19937_64::result_type k_Seed = 3412341234123412; +std::mt19937_64 generator(k_Seed); +std::uniform_real_distribution<> distribution(k_RangeMin, k_RangeMax); // ----------------------------------------------------------------------------- -using VertexType = std::array; -using EdgeType = std::array; - -// ----------------------------------------------------------------------------- -struct VertexHasher -{ - size_t operator()(const VertexType& vert) const - { - size_t hash = std::hash()(vert[0]); - hashCombine(hash, vert[1]); - hashCombine(hash, vert[2]); - return hash; - } -}; - -// ----------------------------------------------------------------------------- -struct EdgeHasher -{ - size_t operator()(const EdgeType& edge) const - { - size_t hash = std::hash()(edge[0]); - hashCombine(hash, edge[1]); - return hash; - } -}; - -// ----------------------------------------------------------------------------- -using VertexMap = std::unordered_map; -using EdgeMap = std::unordered_map; - -// ----------------------------------------------------------------------------- -struct GenerateTripleLinesImpl -{ - using MeshIndexType = typename QuickSurfaceMeshDirect::MeshIndexType; - - GenerateTripleLinesImpl(ImageGeom* imageGeom, Int32AbstractDataStore& featureIdsStore, VertexMap& vertexMapRef, EdgeMap& edgeMapRef) - : origin(imageGeom->getOrigin()) - , res(imageGeom->getSpacing()) - , featureIds(featureIdsStore) - , vertexMap(vertexMapRef) - , edgeMap(edgeMapRef) - { - SizeVec3 udims = imageGeom->getDimensions(); - - xP = udims[0]; - yP = udims[1]; - zP = udims[2]; - } - - void compute(usize zStart, usize zEnd, usize yStart, usize yEnd, usize xStart, usize xEnd) const - { - for(size_t k = zStart; k < zEnd; k++) - { - for(size_t j = yStart; j < yEnd; j++) - { - for(size_t i = xStart; i < xEnd; i++) - { - point = (k * xP * yP) + (j * xP) + i; - // Case 1 - neigh1 = point + 1; - neigh2 = point + (xP * yP) + 1; - neigh3 = point + (xP * yP); - - VertexType const p0 = {{origin[0] + static_cast(i) * res[0] + res[0], origin[1] + static_cast(j) * res[1] + res[1], origin[2] + static_cast(k) * res[2] + res[2]}}; - - VertexType const p1 = {{origin[0] + static_cast(i) * res[0] + res[0], origin[1] + static_cast(j) * res[1], origin[2] + static_cast(k) * res[2] + res[2]}}; - - VertexType const p2 = {{origin[0] + static_cast(i) * res[0], origin[1] + static_cast(j) * res[1] + res[1], origin[2] + static_cast(k) * res[2] + res[2]}}; - - VertexType const p3 = {{origin[0] + static_cast(i) * res[0] + res[0], origin[1] + static_cast(j) * res[1] + res[1], origin[2] + static_cast(k) * res[2]}}; - - uFeatures.clear(); - uFeatures.insert(featureIds[point]); - uFeatures.insert(featureIds[neigh1]); - uFeatures.insert(featureIds[neigh2]); - uFeatures.insert(featureIds[neigh3]); - - if(uFeatures.size() > 2) - { - auto iter = vertexMap.find(p0); - if(iter == vertexMap.end()) - { - vertexMap[p0] = vertCounter++; - } - iter = vertexMap.find(p1); - if(iter == vertexMap.end()) - { - vertexMap[p1] = vertCounter++; - } - MeshIndexType i0 = vertexMap[p0]; - MeshIndexType i1 = vertexMap[p1]; - - const EdgeType tmpEdge = {{i0, i1}}; - auto eiter = edgeMap.find(tmpEdge); - if(eiter == edgeMap.end()) - { - edgeMap[tmpEdge] = edgeCounter++; - } - } - - // Case 2 - neigh1 = point + xP; - neigh2 = point + (xP * yP) + xP; - neigh3 = point + (xP * yP); - - uFeatures.clear(); - uFeatures.insert(featureIds[point]); - uFeatures.insert(featureIds[neigh1]); - uFeatures.insert(featureIds[neigh2]); - uFeatures.insert(featureIds[neigh3]); - if(uFeatures.size() > 2) - { - auto iter = vertexMap.find(p0); - if(iter == vertexMap.end()) - { - vertexMap[p0] = vertCounter++; - } - iter = vertexMap.find(p2); - if(iter == vertexMap.end()) - { - vertexMap[p2] = vertCounter++; - } - - MeshIndexType i0 = vertexMap[p0]; - MeshIndexType i2 = vertexMap[p2]; - - const EdgeType tmpEdge = {{i0, i2}}; - auto eiter = edgeMap.find(tmpEdge); - if(eiter == edgeMap.end()) - { - edgeMap[tmpEdge] = edgeCounter++; - } - } - - // Case 3 - neigh1 = point + 1; - neigh2 = point + xP + 1; - neigh3 = point + +xP; - - uFeatures.clear(); - uFeatures.insert(featureIds[point]); - uFeatures.insert(featureIds[neigh1]); - uFeatures.insert(featureIds[neigh2]); - uFeatures.insert(featureIds[neigh3]); - if(uFeatures.size() > 2) - { - auto iter = vertexMap.find(p0); - if(iter == vertexMap.end()) - { - vertexMap[p0] = vertCounter++; - } - iter = vertexMap.find(p3); - if(iter == vertexMap.end()) - { - vertexMap[p3] = vertCounter++; - } - - MeshIndexType i0 = vertexMap[p0]; - MeshIndexType i3 = vertexMap[p3]; - - const EdgeType tmpEdge = {{i0, i3}}; - auto eiter = edgeMap.find(tmpEdge); - if(eiter == edgeMap.end()) - { - edgeMap[tmpEdge] = edgeCounter++; - } - } - } - } - } - } - - void operator()(const Range3D& range) const - { - compute(range[0], range[1], range[2], range[3], range[4], range[5]); - } - -private: - mutable MeshIndexType point = 0, neigh1 = 0, neigh2 = 0, neigh3 = 0; - mutable MeshIndexType vertCounter = 0; - mutable MeshIndexType edgeCounter = 0; - mutable MeshIndexType xP; - mutable MeshIndexType yP; - mutable MeshIndexType zP; - FloatVec3 origin; - FloatVec3 res; - Int32AbstractDataStore& featureIds; - VertexMap& vertexMap; - EdgeMap& edgeMap; - mutable std::set uFeatures; -}; - -// ----------------------------------------------------------------------------- -void GetGridCoordinates(const IGridGeometry* grid, size_t x, size_t y, size_t z, QuickSurfaceMeshDirect::VertexStore& verts, IGeometry::MeshIndexType nodeIndex) +void GetGridCoordinates(const IGridGeometry* grid, usize x, usize y, usize z, QuickSurfaceMeshDirect::VertexStore& verts, IGeometry::MeshIndexType nodeIndex) { nx::core::Point3D tmpCoords = grid->getPlaneCoords(x, y, z); verts[nodeIndex] = static_cast(tmpCoords[0]); @@ -237,7 +38,7 @@ void GetGridCoordinates(const IGridGeometry* grid, size_t x, size_t y, size_t z, void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3, QuickSurfaceMeshDirect::MeshIndexType v4, QuickSurfaceMeshDirect::MeshIndexType v5, QuickSurfaceMeshDirect::MeshIndexType v6) { - auto val = static_cast(k_Distribution(k_Generator)); // Random remaining position. + auto val = static_cast(distribution(generator)); if(val < 0.25f) { @@ -261,7 +62,7 @@ void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMeshD void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3, QuickSurfaceMeshDirect::MeshIndexType v4) { - auto val = static_cast(k_Distribution(k_Generator)); // Random remaining position. + auto val = static_cast(distribution(generator)); if(val < 0.125f) { @@ -300,7 +101,7 @@ void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMeshD // ----------------------------------------------------------------------------- void FlipProblemVoxelCase3(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3) { - auto val = static_cast(k_Distribution(k_Generator)); // Random remaining position. + auto val = static_cast(distribution(generator)); if(val < 0.5f) { @@ -314,13 +115,14 @@ void FlipProblemVoxelCase3(Int32AbstractDataStore& featureIds, QuickSurfaceMeshD } // namespace // ----------------------------------------------------------------------------- -QuickSurfaceMeshDirect::QuickSurfaceMeshDirect(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler) +QuickSurfaceMeshDirect::QuickSurfaceMeshDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const QuickSurfaceMeshInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) , m_MessageHandler(mesgHandler) { - k_Generator.seed(k_Seed); + generator.seed(k_Seed); } // ----------------------------------------------------------------------------- @@ -329,28 +131,24 @@ QuickSurfaceMeshDirect::~QuickSurfaceMeshDirect() noexcept = default; // ----------------------------------------------------------------------------- Result<> QuickSurfaceMeshDirect::operator()() { - // Get the ImageGeometry auto& grid = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); - - // Get the Created Triangle Geometry auto& triangleGeom = m_DataStructure.getDataRefAs(m_InputValues->TriangleGeometryPath); SizeVec3 udims = grid.getDimensions(); - size_t xP = udims[0]; - size_t yP = udims[1]; - size_t zP = udims[2]; - - std::vector> ownerLists; + usize xP = udims[0]; + usize yP = udims[1]; + usize zP = udims[2]; - size_t possibleNumNodes = (xP + 1) * (yP + 1) * (zP + 1); - std::vector nodeIds(possibleNumNodes, std::numeric_limits::max()); + usize possibleNumNodes = (xP + 1) * (yP + 1) * (zP + 1); + std::vector nodeIds(possibleNumNodes, std::numeric_limits::max()); MeshIndexType nodeCount = 0; MeshIndexType triangleCount = 0; if(m_InputValues->FixProblemVoxels) { + m_MessageHandler(IFilter::Message::Type::Info, "Correcting problem voxels..."); correctProblemVoxels(); } if(m_ShouldCancel) @@ -358,37 +156,35 @@ Result<> QuickSurfaceMeshDirect::operator()() return {}; } + m_MessageHandler(IFilter::Message::Type::Info, "Determining active nodes..."); determineActiveNodes(nodeIds, nodeCount, triangleCount); if(m_ShouldCancel) { return {}; } - // now create node and triangle arrays knowing the number that will be needed ShapeType tupleShape = {triangleCount}; triangleGeom.resizeFaceList(triangleCount); triangleGeom.resizeVertexList(nodeCount); triangleGeom.getFaceAttributeMatrix()->resizeTuples(tupleShape); triangleGeom.getVertexAttributeMatrix()->resizeTuples({nodeCount}); - // Resize the Face Arrays that are being copied over from the ImageGeom Cell Data for(const auto& dataPath : m_InputValues->CreatedDataArrayPaths) { Result<> result = nx::core::ResizeAndReplaceDataArray(m_DataStructure, dataPath, tupleShape, nx::core::IDataAction::Mode::Execute); } + m_MessageHandler(IFilter::Message::Type::Info, "Creating nodes and triangles..."); createNodesAndTriangles(nodeIds, nodeCount, triangleCount); if(m_ShouldCancel) { return {}; } - // Scoped because we invalidate connectivity at the end Result<> windingResult = {}; if(m_InputValues->RepairTriangleWinding) { - // Generate Connectivity - m_MessageHandler("Generating Connectivity and Triangle Neighbors..."); + m_MessageHandler(IFilter::Message::Type::Info, "Generating Connectivity and Triangle Neighbors..."); triangleGeom.findElementNeighbors(true); const auto optionalId = triangleGeom.getElementNeighborsId(); if(!optionalId.has_value()) @@ -397,85 +193,14 @@ Result<> QuickSurfaceMeshDirect::operator()() } const auto& connectivity = m_DataStructure.getDataRefAs(optionalId.value()); - m_MessageHandler("Repairing Windings..."); + m_MessageHandler(IFilter::Message::Type::Info, "Repairing Windings..."); windingResult = MeshingUtilities::RepairTriangleWinding(triangleGeom.getFaces()->getDataStoreRef(), connectivity, m_DataStructure.getDataAs(m_InputValues->FaceLabelsDataPath)->getDataStoreRef(), m_ShouldCancel, m_MessageHandler); - // Purge connectivity m_DataStructure.removeData(triangleGeom.getElementContainingVertId().value()); m_DataStructure.removeData(triangleGeom.getElementNeighborsId().value()); } -#ifdef QSM_CREATE_TRIPLE_LINES - if(m_InputValues->pGenerateTripleLines) - { - IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); - IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); - - EdgeGeom* edgeGeom = EdgeGeom::Create(m_DataStructure, "[EdgeType Geometry]", parentGroupId); - edgeGeom->setVertices(*vertices); - - Int32Array& nodeTypes = m_DataStructure.getDataRefAs(m_InputValues->pFaceLabelsDataPath); - - MeshIndexType edgeCount = 0; - for(MeshIndexType i = 0; i < triangleCount; i++) - { - MeshIndexType n1 = (*triangle)[3 * i + 0]; - MeshIndexType n2 = (*triangle)[3 * i + 1]; - MeshIndexType n3 = (*triangle)[3 * i + 2]; - if(nodeTypes[n1] >= 3 && nodeTypes[n2] >= 3) - { - edgeCount++; - } - if(nodeTypes[n1] >= 3 && nodeTypes[n3] >= 3) - { - edgeCount++; - } - if(nodeTypes[n2] >= 3 && nodeTypes[n3] >= 3) - { - edgeCount++; - } - } - - std::string edgeGeometryName = "[EdgeType Geometry]"; - DataPath edgeGeometryDataPath = m_InputValues->pParentDataGroupPath.createChildPath(edgeGeometryName); - std::string sharedEdgeListName = "SharedEdgeList"; - size_t numEdges = edgeCount; - size_t numEdgeComps = 2; - IGeometry::SharedEdgeList* edges = - IGeometry::SharedEdgeList::CreateWithStore>(m_DataStructure, sharedEdgeListName, {numEdges}, {numEdgeComps}, m_DataStructure.getId(edgeGeometryDataPath)); - - edgeCount = 0; - for(MeshIndexType i = 0; i < triangleCount; i++) - { - MeshIndexType n1 = (*triangle)[3 * i + 0]; - MeshIndexType n2 = (*triangle)[3 * i + 1]; - MeshIndexType n3 = (*triangle)[3 * i + 2]; - if(nodeTypes[n1] >= 3 && nodeTypes[n2] >= 3) - { - (*edges)[2 * edgeCount] = n1; - (*edges)[2 * edgeCount + 1] = n2; - edgeCount++; - } - if(nodeTypes[n1] >= 3 && nodeTypes[n3] >= 3) - { - (*edges)[2 * edgeCount] = n1; - (*edges)[2 * edgeCount + 1] = n3; - edgeCount++; - } - if(nodeTypes[n2] >= 3 && nodeTypes[n3] >= 3) - { - (*edges)[2 * edgeCount] = n2; - (*edges)[2 * edgeCount + 1] = n3; - edgeCount++; - } - } - - // Now that we all of that out of the way, generate the triple lines - generateTripleLines(); - } -#endif - return windingResult; } @@ -496,8 +221,8 @@ void QuickSurfaceMeshDirect::correctProblemVoxels() MeshIndexType v1 = 0, v2 = 0, v3 = 0, v4 = 0; MeshIndexType v5 = 0, v6 = 0, v7 = 0, v8 = 0; - int32_t f1 = 0, f2 = 0, f3 = 0, f4 = 0; - int32_t f5 = 0, f6 = 0, f7 = 0, f8 = 0; + int32 f1 = 0, f2 = 0, f3 = 0, f4 = 0; + int32 f5 = 0, f6 = 0, f7 = 0, f8 = 0; MeshIndexType row1 = 0, row2 = 0; MeshIndexType plane1 = 0, plane2 = 0; @@ -664,14 +389,10 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no MeshIndexType yP = udims[1]; MeshIndexType zP = udims[2]; - std::vector> ownerLists; - MeshIndexType point = 0, neigh1 = 0, neigh2 = 0, neigh3 = 0; MeshIndexType nodeId1 = 0, nodeId2 = 0, nodeId3 = 0, nodeId4 = 0; - // first determining which nodes are actually boundary nodes and - // count number of nodes and triangles that will be created for(MeshIndexType k = 0; k < zP; k++) { if(m_ShouldCancel) @@ -690,25 +411,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no if(i == 0) { nodeId1 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -719,25 +440,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no if(j == 0) { nodeId1 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -748,25 +469,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no if(k == 0) { nodeId1 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -777,25 +498,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no if(i == (xP - 1)) { nodeId1 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -806,25 +527,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no else if(featureIds[point] != featureIds[neigh1]) { nodeId1 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -835,25 +556,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no if(j == (yP - 1)) { nodeId1 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -864,25 +585,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no else if(featureIds[point] != featureIds[neigh2]) { nodeId1 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -893,25 +614,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no if(k == (zP - 1)) { nodeId1 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -922,25 +643,25 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no else if(k < zP - 1 && featureIds[point] != featureIds[neigh3]) { nodeId1 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); - if(nodeIds[nodeId1] == std::numeric_limits::max()) + if(nodeIds[nodeId1] == std::numeric_limits::max()) { nodeIds[nodeId1] = nodeCount; nodeCount++; } nodeId2 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + i; - if(nodeIds[nodeId2] == std::numeric_limits::max()) + if(nodeIds[nodeId2] == std::numeric_limits::max()) { nodeIds[nodeId2] = nodeCount; nodeCount++; } nodeId3 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); - if(nodeIds[nodeId3] == std::numeric_limits::max()) + if(nodeIds[nodeId3] == std::numeric_limits::max()) { nodeIds[nodeId3] = nodeCount; nodeCount++; } nodeId4 = ((k + 1) * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + i; - if(nodeIds[nodeId4] == std::numeric_limits::max()) + if(nodeIds[nodeId4] == std::numeric_limits::max()) { nodeIds[nodeId4] = nodeCount; nodeCount++; @@ -964,14 +685,14 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); - size_t numFeatures = 0; - size_t numTuples = featureIds.getNumberOfTuples(); - for(size_t i = 0; i < numTuples; i++) + usize numFeatures = 0; + usize numTuples = featureIds.getNumberOfTuples(); + for(usize i = 0; i < numTuples; i++) { - const size_t featureId = featureIds[i]; + const usize featureId = featureIds[i]; if(featureId > numFeatures) { - numFeatures = static_cast(featureId); + numFeatures = static_cast(featureId); } } @@ -983,7 +704,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& MeshIndexType yP = udims[1]; MeshIndexType zP = udims[2]; - std::vector> ownerLists; + std::vector> ownerLists; MeshIndexType point = 0; MeshIndexType neigh1 = 0; @@ -997,7 +718,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& auto* triangleGeom = m_DataStructure.getDataAs(m_InputValues->TriangleGeometryPath); - std::vector tDims = {nodeCount}; + ShapeType tDims = {nodeCount}; triangleGeom->resizeVertexList(nodeCount); triangleGeom->resizeFaceList(triangleCount); triangleGeom->getFaceAttributeMatrix()->resizeTuples({triangleCount}); @@ -1005,31 +726,26 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& auto& faceLabelsStore = m_DataStructure.getDataAs(m_InputValues->FaceLabelsDataPath)->getDataStoreRef(); - // Resize the NodeTypes array auto& nodeTypes = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); nodeTypes.resizeTuples({nodeCount}); - QuickSurfaceMeshDirect::VertexStore& vertex = triangleGeom->getVertices()->getDataStoreRef(); - QuickSurfaceMeshDirect::TriStore& triangle = triangleGeom->getFaces()->getDataStoreRef(); + VertexStore& vertex = triangleGeom->getVertices()->getDataStoreRef(); + TriStore& triangle = triangleGeom->getFaces()->getDataStoreRef(); ownerLists.resize(nodeCount); - // Create a vector of TupleTransferFunctions for each of the Triangle Face to VertexType Data Arrays std::vector> tupleTransferFunctions; - for(size_t i = 0; i < m_InputValues->SelectedCellDataArrayPaths.size(); i++) + for(usize i = 0; i < m_InputValues->SelectedCellDataArrayPaths.size(); i++) { - // Associate these arrays with the Triangle Face Data. ::AddTupleTransferInstance(m_DataStructure, m_InputValues->SelectedCellDataArrayPaths[i], m_InputValues->CreatedDataArrayPaths[i], tupleTransferFunctions); } - for(size_t i = 0; i < m_InputValues->SelectedFeatureDataArrayPaths.size(); i++) + for(usize i = 0; i < m_InputValues->SelectedFeatureDataArrayPaths.size(); i++) { - // Associate these arrays with the Triangle Face Data. ::AddFeatureTupleTransferInstance(m_DataStructure, m_InputValues->SelectedFeatureDataArrayPaths[i], m_InputValues->CreatedDataArrayPaths[i + m_InputValues->SelectedCellDataArrayPaths.size()], m_InputValues->FeatureIdsArrayPath, tupleTransferFunctions); } - // Cycle through again assigning coordinates to each node and assigning node numbers and feature labels to each triangle MeshIndexType triangleIndex = 0; for(MeshIndexType k = 0; k < zP; k++) { @@ -1043,7 +759,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& { point = (k * xP * yP) + (j * xP) + i; - neigh1 = point + 1; // <== What happens if we are at the end of a row? + neigh1 = point + 1; neigh2 = point + xP; neigh3 = point + (xP * yP); @@ -1194,7 +910,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& ownerLists[m_NodeIds[nodeId4]].insert(featureIds[point]); ownerLists[m_NodeIds[nodeId4]].insert(-1); } - if(i == (xP - 1)) // Takes care of the end of a Row... + if(i == (xP - 1)) { nodeId1 = (k * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); ::GetGridCoordinates(grid, i + 1, j, k, vertex, (m_NodeIds[nodeId1] * 3)); @@ -1307,7 +1023,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& ownerLists[m_NodeIds[nodeId4]].insert(featureIds[point]); ownerLists[m_NodeIds[nodeId4]].insert(featureIds[neigh1]); } - if(j == (yP - 1)) // Takes care of the end of a column + if(j == (yP - 1)) { nodeId1 = (k * (xP + 1) * (yP + 1)) + ((j + 1) * (xP + 1)) + (i + 1); ::GetGridCoordinates(grid, i + 1, j + 1, k, vertex, (m_NodeIds[nodeId1] * 3)); @@ -1419,7 +1135,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& ownerLists[m_NodeIds[nodeId4]].insert(featureIds[point]); ownerLists[m_NodeIds[nodeId4]].insert(featureIds[neigh2]); } - if(k == (zP - 1)) // Takes care of the end of a Pillar + if(k == (zP - 1)) { nodeId1 = ((k + 1) * (xP + 1) * (yP + 1)) + (j * (xP + 1)) + (i + 1); ::GetGridCoordinates(grid, i + 1, j, k + 1, vertex, (m_NodeIds[nodeId1] * 3)); @@ -1535,13 +1251,20 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& } } + m_MessageHandler(IFilter::Message::Type::Info, "Determining node types..."); + Int8AbstractDataStore& m_NodeTypes = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); - for(size_t i = 0; i < nodeCount; i++) + for(usize i = 0; i < nodeCount; i++) { + if(m_ShouldCancel) + { + return; + } + auto& ownerList = ownerLists[i]; - m_NodeTypes[i] = static_cast(ownerList.size()); + m_NodeTypes[i] = static_cast(ownerList.size()); if(m_NodeTypes[i] > 4) { m_NodeTypes[i] = 4; @@ -1552,226 +1275,3 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& } } } - -// ----------------------------------------------------------------------------- -void QuickSurfaceMeshDirect::generateTripleLines() -{ - if(m_ShouldCancel) - { - return; - } - /** - * This is a bit of experimental code where we define a triple line as an edge - * that shares voxels with at least 3 unique Feature Ids. This is different - * than saying that an edge is part of a triple line if it's nodes are considered - * sharing at least 3 unique voxels. This code is not complete as it will only - * find "interior" triple lines and no lines on the surface. I am going to leave - * this bit of code in place for historical reasons so that we can refer to it - * later if needed. - * Mike Jackson, JULY 2018 - */ - m_MessageHandler(IFilter::Message::Type::Info, "Generating Triple Lines"); - - Int32AbstractDataStore& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); - - int32_t numFeatures = 0; - size_t numTuples = featureIds.getNumberOfTuples(); - for(size_t i = 0; i < numTuples; i++) - { - if(featureIds[i] > numFeatures) - { - numFeatures = featureIds[i]; - } - } - - std::vector featDims = {static_cast(numFeatures) + 1}; - - auto* imageGeom = m_DataStructure.getDataAs(m_InputValues->GridGeomDataPath); - - SizeVec3 udims = imageGeom->getDimensions(); - - MeshIndexType xP = udims[0]; - MeshIndexType yP = udims[1]; - MeshIndexType zP = udims[2]; - MeshIndexType point = 0, neigh1 = 0, neigh2 = 0, neigh3 = 0; - - std::set uFeatures; - - FloatVec3 origin = imageGeom->getOrigin(); - FloatVec3 res = imageGeom->getSpacing(); - - VertexMap vertexMap; - EdgeMap edgeMap; - MeshIndexType vertCounter = 0; - MeshIndexType edgeCounter = 0; - - // Cycle through again assigning coordinates to each node and assigning node numbers and feature labels to each triangle - ParallelData3DAlgorithm algorithm; - algorithm.setRange(Range3D(xP - 1, yP - 1, zP - 1)); - algorithm.setParallelizationEnabled(false); - algorithm.execute(GenerateTripleLinesImpl(imageGeom, featureIds, vertexMap, edgeMap)); - -#if QSM_CREATE_TRIPLE_LINES - for(size_t k = 0; k < zP - 1; k++) - { - for(size_t j = 0; j < yP - 1; j++) - { - for(size_t i = 0; i < xP - 1; i++) - { - point = (k * xP * yP) + (j * xP) + i; - // Case 1 - neigh1 = point + 1; - neigh2 = point + (xP * yP) + 1; - neigh3 = point + (xP * yP); - - VertexType p0 = {{origin[0] + static_cast(i) * res[0] + res[0], origin[1] + static_cast(j) * res[1] + res[1], origin[2] + static_cast(k) * res[2] + res[2]}}; - - VertexType p1 = {{origin[0] + static_cast(i) * res[0] + res[0], origin[1] + static_cast(j) * res[1], origin[2] + static_cast(k) * res[2] + res[2]}}; - - VertexType p2 = {{origin[0] + static_cast(i) * res[0], origin[1] + static_cast(j) * res[1] + res[1], origin[2] + static_cast(k) * res[2] + res[2]}}; - - VertexType p3 = {{origin[0] + static_cast(i) * res[0] + res[0], origin[1] + static_cast(j) * res[1] + res[1], origin[2] + static_cast(k) * res[2]}}; - - uFeatures.clear(); - uFeatures.insert(featureIds[point]); - uFeatures.insert(featureIds[neigh1]); - uFeatures.insert(featureIds[neigh2]); - uFeatures.insert(featureIds[neigh3]); - - if(uFeatures.size() > 2) - { - auto iter = vertexMap.find(p0); - if(iter == vertexMap.end()) - { - vertexMap[p0] = vertCounter++; - } - iter = vertexMap.find(p1); - if(iter == vertexMap.end()) - { - vertexMap[p1] = vertCounter++; - } - MeshIndexType i0 = vertexMap[p0]; - MeshIndexType i1 = vertexMap[p1]; - - EdgeType tmpEdge = {{i0, i1}}; - auto eiter = edgeMap.find(tmpEdge); - if(eiter == edgeMap.end()) - { - edgeMap[tmpEdge] = edgeCounter++; - } - } - - // Case 2 - neigh1 = point + xP; - neigh2 = point + (xP * yP) + xP; - neigh3 = point + (xP * yP); - - uFeatures.clear(); - uFeatures.insert(featureIds[point]); - uFeatures.insert(featureIds[neigh1]); - uFeatures.insert(featureIds[neigh2]); - uFeatures.insert(featureIds[neigh3]); - if(uFeatures.size() > 2) - { - auto iter = vertexMap.find(p0); - if(iter == vertexMap.end()) - { - vertexMap[p0] = vertCounter++; - } - iter = vertexMap.find(p2); - if(iter == vertexMap.end()) - { - vertexMap[p2] = vertCounter++; - } - - MeshIndexType i0 = vertexMap[p0]; - MeshIndexType i2 = vertexMap[p2]; - - EdgeType tmpEdge = {{i0, i2}}; - auto eiter = edgeMap.find(tmpEdge); - if(eiter == edgeMap.end()) - { - edgeMap[tmpEdge] = edgeCounter++; - } - } - - // Case 3 - neigh1 = point + 1; - neigh2 = point + xP + 1; - neigh3 = point + +xP; - - uFeatures.clear(); - uFeatures.insert(featureIds[point]); - uFeatures.insert(featureIds[neigh1]); - uFeatures.insert(featureIds[neigh2]); - uFeatures.insert(featureIds[neigh3]); - if(uFeatures.size() > 2) - { - auto iter = vertexMap.find(p0); - if(iter == vertexMap.end()) - { - vertexMap[p0] = vertCounter++; - } - iter = vertexMap.find(p3); - if(iter == vertexMap.end()) - { - vertexMap[p3] = vertCounter++; - } - - MeshIndexType i0 = vertexMap[p0]; - MeshIndexType i3 = vertexMap[p3]; - - EdgeType tmpEdge = {{i0, i3}}; - auto eiter = edgeMap.find(tmpEdge); - if(eiter == edgeMap.end()) - { - edgeMap[tmpEdge] = edgeCounter++; - } - } - } - } - } -#endif - - std::string edgeGeometryName = "[Edge Geometry]"; - - DataPath edgeGeometryDataPath({edgeGeometryName}); - std::string sharedVertListName = "SharedVertList"; - DataPath sharedVertListDataPath = edgeGeometryDataPath.createChildPath(sharedVertListName); - - EdgeGeom* tripleLineEdge = EdgeGeom::Create(m_DataStructure, edgeGeometryName); - size_t numVerts = vertexMap.size(); - size_t numComps = 3; - IGeometry::SharedVertexList* vertices = Float32Array::CreateWithStore>(m_DataStructure, sharedVertListName, {numVerts}, {numComps}, m_DataStructure.getId(edgeGeometryDataPath)); - auto& verticesRef = vertices->getDataStoreRef(); - - for(const auto& vert : vertexMap) - { - float v0 = vert.first[0]; - float v1 = vert.first[1]; - float v2 = vert.first[2]; - MeshIndexType idx = vert.second; - const MeshIndexType offset = idx * numComps; - verticesRef.setValue(offset + 0, v0); - verticesRef.setValue(+1, v1); - verticesRef.setValue(idx * numComps + 2, v2); - } - tripleLineEdge->setVertices(*vertices); - - std::string sharedEdgeListName = "SharedEdgeList"; - size_t numEdges = edgeMap.size(); - size_t numEdgeComps = 2; - IGeometry::SharedEdgeList* edges = - IGeometry::SharedEdgeList::CreateWithStore>(m_DataStructure, sharedEdgeListName, {numEdges}, {numEdgeComps}, m_DataStructure.getId(edgeGeometryDataPath)); - auto& edgesRef = edges->getDataStoreRef(); - - for(const auto& edge : edgeMap) - { - MeshIndexType i0 = edge.first[0]; - MeshIndexType i1 = edge.first[1]; - MeshIndexType idx = edge.second; - edgesRef.setValue(idx * numComps + 0, i0); - edgesRef.setValue(idx * numComps + 1, i1); - } - tripleLineEdge->setEdgeList(*edges); -} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp index b41e37a472..f9e2388d30 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp @@ -6,32 +6,18 @@ #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/MultiArraySelectionParameter.hpp" - -#include -#include namespace nx::core { - -struct SIMPLNXCORE_EXPORT QuickSurfaceMeshInputValues -{ - bool FixProblemVoxels; - bool RepairTriangleWinding; - bool GenerateTripleLines; - - DataPath GridGeomDataPath; - DataPath FeatureIdsArrayPath; - MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; - MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; - DataPath TriangleGeometryPath; - DataPath VertexGroupDataPath; - DataPath NodeTypesDataPath; - DataPath FaceGroupDataPath; - DataPath FaceLabelsDataPath; - MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; -}; - +struct QuickSurfaceMeshInputValues; + +/** + * @class QuickSurfaceMeshDirect + * @brief In-core algorithm for QuickSurfaceMesh. Preserves the original + * sequential voxel iteration using operator[] on DataStore references. + * Selected by DispatchAlgorithm when all input arrays are backed by + * in-memory DataStore. + */ class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect { public: @@ -39,7 +25,7 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect using TriStore = AbstractDataStore; using MeshIndexType = IGeometry::MeshIndexType; - QuickSurfaceMeshDirect(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler); + QuickSurfaceMeshDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const QuickSurfaceMeshInputValues* inputValues); ~QuickSurfaceMeshDirect() noexcept; QuickSurfaceMeshDirect(const QuickSurfaceMeshDirect&) = delete; @@ -49,37 +35,15 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect Result<> operator()(); - /** - * @brief - */ +private: void correctProblemVoxels(); + void determineActiveNodes(std::vector& nodeIds, MeshIndexType& nodeCount, MeshIndexType& triangleCount); + void createNodesAndTriangles(std::vector& nodeIds, MeshIndexType nodeCount, MeshIndexType triangleCount); - /** - * @brief - * @param m_NodeIds - * @param nodeCount - * @param triangleCount - */ - void determineActiveNodes(std::vector& m_NodeIds, MeshIndexType& nodeCount, MeshIndexType& triangleCount); - - /** - * @brief - * @param m_NodeIds - * @param nodeCount - * @param triangleCount - */ - void createNodesAndTriangles(std::vector& m_NodeIds, MeshIndexType nodeCount, MeshIndexType triangleCount); - - /** - * @brief generateTripleLines - */ - void generateTripleLines(); - -private: DataStructure& m_DataStructure; const QuickSurfaceMeshInputValues* m_InputValues = nullptr; const std::atomic_bool& m_ShouldCancel; const IFilter::MessageHandler& m_MessageHandler; - bool m_GenerateTripleLines = false; }; + } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp new file mode 100644 index 0000000000..4e0a829b40 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp @@ -0,0 +1,1326 @@ +#include "QuickSurfaceMeshScanline.hpp" + +#include "QuickSurfaceMesh.hpp" +#include "TupleTransfer.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/EdgeGeom.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/Geometry/TriangleGeom.hpp" +#include "simplnx/Utilities/DataArrayUtilities.hpp" +#include "simplnx/Utilities/Meshing/TriangleUtilities.hpp" + +#include +#include +#include +#include + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +namespace +{ +constexpr float64 k_RangeMin = 0.0; +constexpr float64 k_RangeMax = 1.0; +constexpr std::mt19937_64::result_type k_Seed = 3412341234123412; +std::mt19937_64 generator(k_Seed); +std::uniform_real_distribution<> distribution(k_RangeMin, k_RangeMax); + +// Buffer-based flip functions that operate on raw int32 pointers +// ----------------------------------------------------------------------------- +void FlipProblemVoxelCase1(int32* buf, QuickSurfaceMeshScanline::MeshIndexType v1, QuickSurfaceMeshScanline::MeshIndexType v2, QuickSurfaceMeshScanline::MeshIndexType v3, + QuickSurfaceMeshScanline::MeshIndexType v4, QuickSurfaceMeshScanline::MeshIndexType v5, QuickSurfaceMeshScanline::MeshIndexType v6) +{ + auto val = static_cast(distribution(generator)); + + if(val < 0.25f) + { + buf[v6] = buf[v4]; + } + else if(val < 0.5f) + { + buf[v6] = buf[v5]; + } + else if(val < 0.75f) + { + buf[v1] = buf[v2]; + } + else + { + buf[v1] = buf[v3]; + } +} + +// ----------------------------------------------------------------------------- +void FlipProblemVoxelCase2(int32* buf, QuickSurfaceMeshScanline::MeshIndexType v1, QuickSurfaceMeshScanline::MeshIndexType v2, QuickSurfaceMeshScanline::MeshIndexType v3, + QuickSurfaceMeshScanline::MeshIndexType v4) +{ + auto val = static_cast(distribution(generator)); + + if(val < 0.125f) + { + buf[v1] = buf[v2]; + } + else if(val < 0.25f) + { + buf[v1] = buf[v3]; + } + else if(val < 0.375f) + { + buf[v2] = buf[v1]; + } + if(val < 0.5f) + { + buf[v2] = buf[v4]; + } + else if(val < 0.625f) + { + buf[v3] = buf[v1]; + } + else if(val < 0.75f) + { + buf[v3] = buf[v4]; + } + else if(val < 0.875f) + { + buf[v4] = buf[v2]; + } + else + { + buf[v4] = buf[v3]; + } +} + +// ----------------------------------------------------------------------------- +void FlipProblemVoxelCase3(int32* buf, QuickSurfaceMeshScanline::MeshIndexType v1, QuickSurfaceMeshScanline::MeshIndexType v2, QuickSurfaceMeshScanline::MeshIndexType v3) +{ + auto val = static_cast(distribution(generator)); + + if(val < 0.5f) + { + buf[v2] = buf[v1]; + } + else + { + buf[v3] = buf[v1]; + } +} +} // namespace + +// ----------------------------------------------------------------------------- +QuickSurfaceMeshScanline::QuickSurfaceMeshScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, + const QuickSurfaceMeshInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ + generator.seed(k_Seed); +} + +// ----------------------------------------------------------------------------- +QuickSurfaceMeshScanline::~QuickSurfaceMeshScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> QuickSurfaceMeshScanline::operator()() +{ + auto& grid = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); + auto& triangleGeom = m_DataStructure.getDataRefAs(m_InputValues->TriangleGeometryPath); + + SizeVec3 udims = grid.getDimensions(); + + usize xP = udims[0]; + usize yP = udims[1]; + usize zP = udims[2]; + + MeshIndexType nodeCount = 0; + MeshIndexType triangleCount = 0; + usize numFeatures = 0; + + if(m_InputValues->FixProblemVoxels) + { + correctProblemVoxels(); + } + if(m_ShouldCancel) + { + return {}; + } + + countActiveNodesAndTriangles(nodeCount, triangleCount, numFeatures); + if(m_ShouldCancel) + { + return {}; + } + + ShapeType tupleShape = {triangleCount}; + triangleGeom.resizeFaceList(triangleCount); + triangleGeom.resizeVertexList(nodeCount); + triangleGeom.getFaceAttributeMatrix()->resizeTuples(tupleShape); + triangleGeom.getVertexAttributeMatrix()->resizeTuples({nodeCount}); + + for(const auto& dataPath : m_InputValues->CreatedDataArrayPaths) + { + Result<> result = nx::core::ResizeAndReplaceDataArray(m_DataStructure, dataPath, tupleShape, nx::core::IDataAction::Mode::Execute); + } + + createNodesAndTriangles(nodeCount, triangleCount, numFeatures); + if(m_ShouldCancel) + { + return {}; + } + + Result<> windingResult = {}; + if(m_InputValues->RepairTriangleWinding) + { + triangleGeom.findElementNeighbors(true); + const auto optionalId = triangleGeom.getElementNeighborsId(); + if(!optionalId.has_value()) + { + return MakeErrorResult(-56341, fmt::format("Unable to generate the connectivity list for {} geometry.", triangleGeom.getName())); + } + const auto& connectivity = m_DataStructure.getDataRefAs(optionalId.value()); + + windingResult = MeshingUtilities::RepairTriangleWinding(triangleGeom.getFaces()->getDataStoreRef(), connectivity, + m_DataStructure.getDataAs(m_InputValues->FaceLabelsDataPath)->getDataStoreRef(), m_ShouldCancel, m_MessageHandler); + + m_DataStructure.removeData(triangleGeom.getElementContainingVertId().value()); + m_DataStructure.removeData(triangleGeom.getElementNeighborsId().value()); + } + + return windingResult; +} + +// ----------------------------------------------------------------------------- +void QuickSurfaceMeshScanline::correctProblemVoxels() +{ + m_MessageHandler(IFilter::Message::Type::Info, "Correcting Problem Voxels"); + + auto* grid = m_DataStructure.getDataAs(m_InputValues->GridGeomDataPath); + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + + SizeVec3 udims = grid->getDimensions(); + + MeshIndexType xP = udims[0]; + MeshIndexType yP = udims[1]; + MeshIndexType zP = udims[2]; + + const MeshIndexType sliceSize = xP * yP; + + // Buffer two consecutive z-slices at a time + auto sliceA = std::make_unique(sliceSize); + auto sliceB = std::make_unique(sliceSize); + + MeshIndexType count = 1; + MeshIndexType iter = 0; + while(count > 0 && iter < 20) + { + if(m_ShouldCancel) + { + return; + } + iter++; + count = 0; + + for(MeshIndexType k = 1; k < zP; k++) + { + // Load slice (k-1) into sliceA and slice k into sliceB + featureIdsStore.copyIntoBuffer((k - 1) * sliceSize, nonstd::span(sliceA.get(), sliceSize)); + featureIdsStore.copyIntoBuffer(k * sliceSize, nonstd::span(sliceB.get(), sliceSize)); + + bool sliceADirty = false; + bool sliceBDirty = false; + + for(MeshIndexType j = 1; j < yP; j++) + { + MeshIndexType row1 = (j - 1) * xP; + MeshIndexType row2 = j * xP; + for(MeshIndexType i = 1; i < xP; i++) + { + // v1-v4 are in slice (k-1) = sliceA, v5-v8 are in slice k = sliceB + MeshIndexType v1Local = row1 + i - 1; + MeshIndexType v2Local = row1 + i; + MeshIndexType v3Local = row2 + i - 1; + MeshIndexType v4Local = row2 + i; + + int32 f1 = sliceA[v1Local]; + int32 f2 = sliceA[v2Local]; + int32 f3 = sliceA[v3Local]; + int32 f4 = sliceA[v4Local]; + int32 f5 = sliceB[v1Local]; + int32 f6 = sliceB[v2Local]; + int32 f7 = sliceB[v3Local]; + int32 f8 = sliceB[v4Local]; + + // For the flip functions, we need indices into a combined 2-slice buffer. + // sliceA occupies [0, sliceSize), sliceB occupies [sliceSize, 2*sliceSize) + // But since FlipProblemVoxelCase functions operate on voxels that may be + // in either slice, we use a combined buffer approach: + // We'll use local indices directly into the correct slice buffer. + + if(f1 == f8 && f1 != f2 && f1 != f3 && f1 != f4 && f1 != f5 && f1 != f6 && f1 != f7) + { + // v1=plane1+row1+i-1, v2=plane1+row1+i, v3=plane1+row2+i-1 + // v6=plane2+row1+i, v7=plane2+row2+i-1, v8=plane2+row2+i + // FlipProblemVoxelCase1(featureIds, v1, v2, v3, v6, v7, v8) + auto val = static_cast(distribution(generator)); + if(val < 0.25f) + { + sliceB[v4Local] = sliceB[v2Local]; // v8 = v6 + sliceBDirty = true; + } + else if(val < 0.5f) + { + sliceB[v4Local] = sliceB[v3Local]; // v8 = v7 + sliceBDirty = true; + } + else if(val < 0.75f) + { + sliceA[v1Local] = sliceA[v2Local]; // v1 = v2 + sliceADirty = true; + } + else + { + sliceA[v1Local] = sliceA[v3Local]; // v1 = v3 + sliceADirty = true; + } + count++; + } + if(f2 == f7 && f2 != f1 && f2 != f3 && f2 != f4 && f2 != f5 && f2 != f6 && f2 != f8) + { + // FlipProblemVoxelCase1(featureIds, v2, v1, v4, v5, v8, v7) + auto val = static_cast(distribution(generator)); + if(val < 0.25f) + { + sliceB[v3Local] = sliceB[v1Local]; // v7 = v5 + sliceBDirty = true; + } + else if(val < 0.5f) + { + sliceB[v3Local] = sliceB[v4Local]; // v7 = v8 + sliceBDirty = true; + } + else if(val < 0.75f) + { + sliceA[v2Local] = sliceA[v1Local]; // v2 = v1 + sliceADirty = true; + } + else + { + sliceA[v2Local] = sliceA[v4Local]; // v2 = v4 + sliceADirty = true; + } + count++; + } + if(f3 == f6 && f3 != f1 && f3 != f2 && f3 != f4 && f3 != f5 && f3 != f7 && f3 != f8) + { + // FlipProblemVoxelCase1(featureIds, v3, v1, v4, v5, v8, v6) + auto val = static_cast(distribution(generator)); + if(val < 0.25f) + { + sliceB[v2Local] = sliceB[v1Local]; // v6 = v5 + sliceBDirty = true; + } + else if(val < 0.5f) + { + sliceB[v2Local] = sliceB[v4Local]; // v6 = v8 + sliceBDirty = true; + } + else if(val < 0.75f) + { + sliceA[v3Local] = sliceA[v1Local]; // v3 = v1 + sliceADirty = true; + } + else + { + sliceA[v3Local] = sliceA[v4Local]; // v3 = v4 + sliceADirty = true; + } + count++; + } + if(f4 == f5 && f4 != f1 && f4 != f2 && f4 != f3 && f4 != f6 && f4 != f7 && f4 != f8) + { + // FlipProblemVoxelCase1(featureIds, v4, v2, v3, v6, v7, v5) + auto val = static_cast(distribution(generator)); + if(val < 0.25f) + { + sliceB[v1Local] = sliceB[v2Local]; // v5 = v6 + sliceBDirty = true; + } + else if(val < 0.5f) + { + sliceB[v1Local] = sliceB[v3Local]; // v5 = v7 + sliceBDirty = true; + } + else if(val < 0.75f) + { + sliceA[v4Local] = sliceA[v2Local]; // v4 = v2 + sliceADirty = true; + } + else + { + sliceA[v4Local] = sliceA[v3Local]; // v4 = v3 + sliceADirty = true; + } + count++; + } + + // Case2 variants - these use FlipProblemVoxelCase2 which operates on 4 voxels + // We inline the RNG consumption but delegate to a helper for the actual mutation + auto doCase2 = [&](int32* bufX, MeshIndexType ix1, int32* bufY, MeshIndexType iy1, int32* bufZ, MeshIndexType iz1, int32* bufW, MeshIndexType iw1, bool& dirtyX, bool& dirtyY, bool& dirtyZ, + bool& dirtyW) { + auto val = static_cast(distribution(generator)); + if(val < 0.125f) + { + bufX[ix1] = bufY[iy1]; + dirtyX = true; + } + else if(val < 0.25f) + { + bufX[ix1] = bufZ[iz1]; + dirtyX = true; + } + else if(val < 0.375f) + { + bufY[iy1] = bufX[ix1]; + dirtyY = true; + } + if(val < 0.5f) + { + bufY[iy1] = bufW[iw1]; + dirtyY = true; + } + else if(val < 0.625f) + { + bufZ[iz1] = bufX[ix1]; + dirtyZ = true; + } + else if(val < 0.75f) + { + bufZ[iz1] = bufW[iw1]; + dirtyZ = true; + } + else if(val < 0.875f) + { + bufW[iw1] = bufY[iy1]; + dirtyW = true; + } + else + { + bufW[iw1] = bufZ[iz1]; + dirtyW = true; + } + }; + + // f1==f6: v1(A),v2(A),v5(B),v6(B) + if(f1 == f6 && f1 != f2 && f1 != f5) + { + doCase2(sliceA.get(), v1Local, sliceA.get(), v2Local, sliceB.get(), v1Local, sliceB.get(), v2Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f2 == f5 && f2 != f1 && f2 != f6) + { + doCase2(sliceA.get(), v2Local, sliceA.get(), v1Local, sliceB.get(), v2Local, sliceB.get(), v1Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f3 == f8 && f3 != f4 && f3 != f7) + { + doCase2(sliceA.get(), v3Local, sliceA.get(), v4Local, sliceB.get(), v3Local, sliceB.get(), v4Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f4 == f7 && f4 != f3 && f4 != f8) + { + doCase2(sliceA.get(), v4Local, sliceA.get(), v3Local, sliceB.get(), v4Local, sliceB.get(), v3Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f1 == f7 && f1 != f3 && f1 != f5) + { + doCase2(sliceA.get(), v1Local, sliceA.get(), v3Local, sliceB.get(), v1Local, sliceB.get(), v3Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f3 == f5 && f3 != f1 && f3 != f7) + { + doCase2(sliceA.get(), v3Local, sliceA.get(), v1Local, sliceB.get(), v3Local, sliceB.get(), v1Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f2 == f8 && f2 != f4 && f2 != f6) + { + doCase2(sliceA.get(), v2Local, sliceA.get(), v4Local, sliceB.get(), v2Local, sliceB.get(), v4Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + if(f4 == f6 && f4 != f2 && f4 != f8) + { + doCase2(sliceA.get(), v4Local, sliceA.get(), v2Local, sliceB.get(), v4Local, sliceB.get(), v2Local, sliceADirty, sliceADirty, sliceBDirty, sliceBDirty); + count++; + } + // Same-plane Case2 variants (all in sliceA or all in sliceB) + if(f1 == f4 && f1 != f2 && f1 != f3) + { + doCase2(sliceA.get(), v1Local, sliceA.get(), v2Local, sliceA.get(), v3Local, sliceA.get(), v4Local, sliceADirty, sliceADirty, sliceADirty, sliceADirty); + count++; + } + if(f2 == f3 && f2 != f1 && f2 != f4) + { + doCase2(sliceA.get(), v2Local, sliceA.get(), v1Local, sliceA.get(), v4Local, sliceA.get(), v3Local, sliceADirty, sliceADirty, sliceADirty, sliceADirty); + count++; + } + if(f5 == f8 && f5 != f6 && f5 != f7) + { + doCase2(sliceB.get(), v1Local, sliceB.get(), v2Local, sliceB.get(), v3Local, sliceB.get(), v4Local, sliceBDirty, sliceBDirty, sliceBDirty, sliceBDirty); + count++; + } + if(f6 == f7 && f6 != f5 && f6 != f8) + { + doCase2(sliceB.get(), v2Local, sliceB.get(), v1Local, sliceB.get(), v4Local, sliceB.get(), v3Local, sliceBDirty, sliceBDirty, sliceBDirty, sliceBDirty); + count++; + } + + // Case3 variants + if(f2 == f3 && f2 == f4 && f2 == f5 && f2 == f6 && f2 == f7 && f2 != f1 && f2 != f8) + { + auto val = static_cast(distribution(generator)); + if(val < 0.5f) + { + sliceA[v1Local] = sliceA[v2Local]; // v1 = v2 + sliceADirty = true; + } + else + { + sliceB[v4Local] = sliceA[v2Local]; // v8 = v2 + sliceBDirty = true; + } + count++; + } + if(f1 == f3 && f1 == f4 && f1 == f5 && f1 == f7 && f2 == f8 && f1 != f2 && f1 != f7) + { + auto val = static_cast(distribution(generator)); + if(val < 0.5f) + { + sliceA[v2Local] = sliceA[v1Local]; // v2 = v1 + sliceADirty = true; + } + else + { + sliceB[v3Local] = sliceA[v1Local]; // v7 = v1 + sliceBDirty = true; + } + count++; + } + if(f1 == f2 && f1 == f4 && f1 == f5 && f1 == f7 && f1 == f8 && f1 != f3 && f1 != f6) + { + auto val = static_cast(distribution(generator)); + if(val < 0.5f) + { + sliceA[v3Local] = sliceA[v1Local]; // v3 = v1 + sliceADirty = true; + } + else + { + sliceB[v2Local] = sliceA[v1Local]; // v6 = v1 + sliceBDirty = true; + } + count++; + } + if(f1 == f2 && f1 == f3 && f1 == f6 && f1 == f7 && f1 == f8 && f1 != f4 && f1 != f5) + { + auto val = static_cast(distribution(generator)); + if(val < 0.5f) + { + sliceA[v4Local] = sliceA[v1Local]; // v4 = v1 + sliceADirty = true; + } + else + { + sliceB[v1Local] = sliceA[v1Local]; // v5 = v1 + sliceBDirty = true; + } + count++; + } + } + } + + // Write back dirty slices + if(sliceADirty) + { + featureIdsStore.copyFromBuffer((k - 1) * sliceSize, nonstd::span(sliceA.get(), sliceSize)); + } + if(sliceBDirty) + { + featureIdsStore.copyFromBuffer(k * sliceSize, nonstd::span(sliceB.get(), sliceSize)); + } + } + + std::string ss = fmt::format("Correcting Problem Voxels: Iteration - '{}'; Problem Voxels - '{}'", iter, count); + m_MessageHandler(IFilter::Message::Type::Info, ss); + } +} + +// ----------------------------------------------------------------------------- +void QuickSurfaceMeshScanline::countActiveNodesAndTriangles(MeshIndexType& nodeCount, MeshIndexType& triangleCount, usize& numFeatures) +{ + m_MessageHandler(IFilter::Message::Type::Info, "Counting active nodes and triangles"); + + auto* grid = m_DataStructure.getDataAs(m_InputValues->GridGeomDataPath); + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + + SizeVec3 udims = grid->getDimensions(); + + MeshIndexType xP = udims[0]; + MeshIndexType yP = udims[1]; + MeshIndexType zP = udims[2]; + + const MeshIndexType sliceSize = xP * yP; + const MeshIndexType nodePlaneSize = (xP + 1) * (yP + 1); + constexpr auto kMax = std::numeric_limits::max(); + + // Rolling node-plane buffers: O(2 * nodePlaneSize) instead of O((xP+1)*(yP+1)*(zP+1)) + std::vector nodePlane0(nodePlaneSize, kMax); + std::vector nodePlane1(nodePlaneSize, kMax); + + // Lambda to count a node: if not yet assigned, assign and increment + auto countNode = [&](std::vector& plane, MeshIndexType offset) { + if(plane[offset] == kMax) + { + plane[offset] = nodeCount; + nodeCount++; + } + }; + + // Buffer current and next z-slices for featureIds + auto curSlice = std::make_unique(sliceSize); + auto nextSlice = std::make_unique(sliceSize); + + // Load first slice + featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.get(), sliceSize)); + + numFeatures = 0; + + for(MeshIndexType k = 0; k < zP; k++) + { + if(m_ShouldCancel) + { + return; + } + // Load next z-slice if available + if(k < zP - 1) + { + featureIdsStore.copyIntoBuffer((k + 1) * sliceSize, nonstd::span(nextSlice.get(), sliceSize)); + } + + for(MeshIndexType j = 0; j < yP; j++) + { + for(MeshIndexType i = 0; i < xP; i++) + { + MeshIndexType localIdx = j * xP + i; + int32 curFeature = curSlice[localIdx]; + + // Track max featureId for numFeatures + if(static_cast(curFeature) > numFeatures) + { + numFeatures = static_cast(curFeature); + } + + // Node offsets within a plane for node grid position (ni, nj): + // offset = nj * (xP + 1) + ni + // Plane 0 corresponds to z=k, Plane 1 corresponds to z=k+1 + + if(i == 0) + { + countNode(nodePlane0, j * (xP + 1) + i); + countNode(nodePlane0, (j + 1) * (xP + 1) + i); + countNode(nodePlane1, j * (xP + 1) + i); + countNode(nodePlane1, (j + 1) * (xP + 1) + i); + triangleCount += 2; + } + if(j == 0) + { + countNode(nodePlane0, j * (xP + 1) + i); + countNode(nodePlane0, j * (xP + 1) + (i + 1)); + countNode(nodePlane1, j * (xP + 1) + i); + countNode(nodePlane1, j * (xP + 1) + (i + 1)); + triangleCount += 2; + } + if(k == 0) + { + countNode(nodePlane0, j * (xP + 1) + i); + countNode(nodePlane0, j * (xP + 1) + (i + 1)); + countNode(nodePlane0, (j + 1) * (xP + 1) + i); + countNode(nodePlane0, (j + 1) * (xP + 1) + (i + 1)); + triangleCount += 2; + } + if(i == (xP - 1)) + { + countNode(nodePlane0, j * (xP + 1) + (i + 1)); + countNode(nodePlane0, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane1, j * (xP + 1) + (i + 1)); + countNode(nodePlane1, (j + 1) * (xP + 1) + (i + 1)); + triangleCount += 2; + } + else if(curFeature != curSlice[localIdx + 1]) // neigh1 = point + 1 + { + countNode(nodePlane0, j * (xP + 1) + (i + 1)); + countNode(nodePlane0, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane1, j * (xP + 1) + (i + 1)); + countNode(nodePlane1, (j + 1) * (xP + 1) + (i + 1)); + triangleCount += 2; + } + if(j == (yP - 1)) + { + countNode(nodePlane0, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane0, (j + 1) * (xP + 1) + i); + countNode(nodePlane1, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane1, (j + 1) * (xP + 1) + i); + triangleCount += 2; + } + else if(curFeature != curSlice[localIdx + xP]) // neigh2 = point + xP + { + countNode(nodePlane0, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane0, (j + 1) * (xP + 1) + i); + countNode(nodePlane1, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane1, (j + 1) * (xP + 1) + i); + triangleCount += 2; + } + if(k == (zP - 1)) + { + countNode(nodePlane1, j * (xP + 1) + (i + 1)); + countNode(nodePlane1, j * (xP + 1) + i); + countNode(nodePlane1, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane1, (j + 1) * (xP + 1) + i); + triangleCount += 2; + } + else if(curFeature != nextSlice[localIdx]) // neigh3 = point + xP*yP + { + countNode(nodePlane1, j * (xP + 1) + (i + 1)); + countNode(nodePlane1, j * (xP + 1) + i); + countNode(nodePlane1, (j + 1) * (xP + 1) + (i + 1)); + countNode(nodePlane1, (j + 1) * (xP + 1) + i); + triangleCount += 2; + } + } + } + + // Rotate planes: plane1 becomes plane0 for next z-step, reinitialize plane1 + std::swap(nodePlane0, nodePlane1); + std::fill(nodePlane1.begin(), nodePlane1.end(), kMax); + + // Swap featureId buffers: current becomes the old "next" + std::swap(curSlice, nextSlice); + } +} + +// ----------------------------------------------------------------------------- +void QuickSurfaceMeshScanline::createNodesAndTriangles(MeshIndexType nodeCount, MeshIndexType triangleCount, usize numFeatures) +{ + if(m_ShouldCancel) + { + return; + } + m_MessageHandler(IFilter::Message::Type::Info, "Creating mesh"); + + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + + auto* grid = m_DataStructure.getDataAs(m_InputValues->GridGeomDataPath); + + SizeVec3 udims = grid->getDimensions(); + + MeshIndexType xP = udims[0]; + MeshIndexType yP = udims[1]; + MeshIndexType zP = udims[2]; + + const MeshIndexType sliceSize = xP * yP; + const MeshIndexType nodePlaneSize = (xP + 1) * (yP + 1); + constexpr auto kMax = std::numeric_limits::max(); + + auto* triangleGeom = m_DataStructure.getDataAs(m_InputValues->TriangleGeometryPath); + + ShapeType tDims = {nodeCount}; + triangleGeom->resizeVertexList(nodeCount); + triangleGeom->resizeFaceList(triangleCount); + triangleGeom->getFaceAttributeMatrix()->resizeTuples({triangleCount}); + triangleGeom->getVertexAttributeMatrix()->resizeTuples(tDims); + + auto& faceLabelsStore = m_DataStructure.getDataAs(m_InputValues->FaceLabelsDataPath)->getDataStoreRef(); + + auto& nodeTypesStore = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); + nodeTypesStore.resizeTuples({nodeCount}); + + VertexStore& vertex = triangleGeom->getVertices()->getDataStoreRef(); + TriStore& triangle = triangleGeom->getFaces()->getDataStoreRef(); + + std::vector> ownerLists(nodeCount); + + std::vector> tupleTransferFunctions; + for(usize i = 0; i < m_InputValues->SelectedCellDataArrayPaths.size(); i++) + { + ::AddTupleTransferInstance(m_DataStructure, m_InputValues->SelectedCellDataArrayPaths[i], m_InputValues->CreatedDataArrayPaths[i], tupleTransferFunctions); + } + + for(usize i = 0; i < m_InputValues->SelectedFeatureDataArrayPaths.size(); i++) + { + ::AddFeatureTupleTransferInstance(m_DataStructure, m_InputValues->SelectedFeatureDataArrayPaths[i], m_InputValues->CreatedDataArrayPaths[i + m_InputValues->SelectedCellDataArrayPaths.size()], + m_InputValues->FeatureIdsArrayPath, tupleTransferFunctions); + } + + // Buffer current and next z-slices for featureIds + auto curSlice = std::make_unique(sliceSize); + auto nextSlice = std::make_unique(sliceSize); + + // Rolling node-plane buffers: O(2 * nodePlaneSize) instead of O((xP+1)*(yP+1)*(zP+1)) + std::vector nodePlane0(nodePlaneSize, kMax); + std::vector nodePlane1(nodePlaneSize, kMax); + + // Buffer all vertex coordinates in memory (O(surface_area), not O(volume)), + // flushed once at the end to avoid per-element OOC writes. + auto vertCoordBuf = std::make_unique(nodeCount * 3); + + // Lambda to assign a node: if not yet assigned, assign sequential ID. + // Always write vertex coordinates into the local buffer (last-write-wins + // matches original behavior where multiple calls may pass different coords). + auto assignNode = [&](std::vector& plane, MeshIndexType offset, MeshIndexType& assignedNodeCount, usize coordX, usize coordY, usize coordZ) { + if(plane[offset] == kMax) + { + plane[offset] = assignedNodeCount; + assignedNodeCount++; + } + nx::core::Point3D tmpCoords = grid->getPlaneCoords(coordX, coordY, coordZ); + MeshIndexType vi = plane[offset] * 3; + vertCoordBuf[vi] = static_cast(tmpCoords[0]); + vertCoordBuf[vi + 1] = static_cast(tmpCoords[1]); + vertCoordBuf[vi + 2] = static_cast(tmpCoords[2]); + }; + + // Load first slice + featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.get(), sliceSize)); + + MeshIndexType triangleIndex = 0; + MeshIndexType assignedNodeCount = 0; + + // Per-slice buffers declared outside the loop so vector capacity is reused across slices. + std::vector triBuffer; + std::vector faceLabelBuf; + std::vector ttArgsBuf; + + for(MeshIndexType k = 0; k < zP; k++) + { + if(m_ShouldCancel) + { + return; + } + // Load next z-slice if available + if(k < zP - 1) + { + featureIdsStore.copyIntoBuffer((k + 1) * sliceSize, nonstd::span(nextSlice.get(), sliceSize)); + } + + triBuffer.clear(); + faceLabelBuf.clear(); + ttArgsBuf.clear(); + + for(MeshIndexType j = 0; j < yP; j++) + { + for(MeshIndexType i = 0; i < xP; i++) + { + MeshIndexType localIdx = j * xP + i; + MeshIndexType point = k * sliceSize + localIdx; + int32 curFeature = curSlice[localIdx]; + + // Node plane offsets: offset = nj * (xP+1) + ni + // nodePlane0 = z=k plane, nodePlane1 = z=k+1 plane + + if(i == 0) + { + MeshIndexType n1Off = j * (xP + 1) + i; + MeshIndexType n2Off = (j + 1) * (xP + 1) + i; + MeshIndexType n3Off = j * (xP + 1) + i; + MeshIndexType n4Off = (j + 1) * (xP + 1) + i; + assignNode(nodePlane0, n1Off, assignedNodeCount, i, j, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i, j + 1, k); + assignNode(nodePlane1, n3Off, assignedNodeCount, i, j, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i + 1, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + triBuffer.push_back(nid3); + triBuffer.push_back(nid2); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + triBuffer.push_back(nid4); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(-1); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(-1); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(-1); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(-1); + } + if(j == 0) + { + MeshIndexType n1Off = j * (xP + 1) + i; + MeshIndexType n2Off = j * (xP + 1) + (i + 1); + MeshIndexType n3Off = j * (xP + 1) + i; + MeshIndexType n4Off = j * (xP + 1) + (i + 1); + assignNode(nodePlane0, n1Off, assignedNodeCount, i, j, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i + 1, j, k); + assignNode(nodePlane1, n3Off, assignedNodeCount, i, j, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i + 1, j, k + 1); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + triBuffer.push_back(nid2); + triBuffer.push_back(nid4); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(-1); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(-1); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(-1); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(-1); + } + if(k == 0) + { + MeshIndexType n1Off = j * (xP + 1) + i; + MeshIndexType n2Off = j * (xP + 1) + (i + 1); + MeshIndexType n3Off = (j + 1) * (xP + 1) + i; + MeshIndexType n4Off = (j + 1) * (xP + 1) + (i + 1); + assignNode(nodePlane0, n1Off, assignedNodeCount, i, j, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i + 1, j, k); + assignNode(nodePlane0, n3Off, assignedNodeCount, i, j + 1, k); + assignNode(nodePlane0, n4Off, assignedNodeCount, i + 1, j + 1, k); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane0[n3Off]; + MeshIndexType nid4 = nodePlane0[n4Off]; + + triBuffer.push_back(nid1); + triBuffer.push_back(nid3); + triBuffer.push_back(nid2); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + triBuffer.push_back(nid4); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(-1); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(-1); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(-1); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(-1); + } + if(i == (xP - 1)) + { + MeshIndexType n1Off = j * (xP + 1) + (i + 1); + MeshIndexType n2Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n3Off = j * (xP + 1) + (i + 1); + MeshIndexType n4Off = (j + 1) * (xP + 1) + (i + 1); + assignNode(nodePlane0, n1Off, assignedNodeCount, i + 1, j, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i + 1, j + 1, k); + assignNode(nodePlane1, n3Off, assignedNodeCount, i + 1, j, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i + 1, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + triBuffer.push_back(nid2); + triBuffer.push_back(nid4); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(-1); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(-1); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(-1); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(-1); + } + else if(curFeature != curSlice[localIdx + 1]) + { + int32 neigh1Feature = curSlice[localIdx + 1]; + MeshIndexType neigh1 = point + 1; + + MeshIndexType n1Off = j * (xP + 1) + (i + 1); + MeshIndexType n2Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n3Off = j * (xP + 1) + (i + 1); + MeshIndexType n4Off = (j + 1) * (xP + 1) + (i + 1); + assignNode(nodePlane0, n1Off, assignedNodeCount, i + 1, j, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i + 1, j + 1, k); + assignNode(nodePlane1, n3Off, assignedNodeCount, i + 1, j, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i + 1, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + if(curFeature < neigh1Feature) + { + triBuffer.push_back(nid3); + triBuffer.push_back(nid2); + faceLabelBuf.push_back(curFeature); + faceLabelBuf.push_back(neigh1Feature); + ttArgsBuf.push_back({triangleIndex, neigh1, point, curFeature, neigh1Feature}); + } + else + { + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(neigh1Feature); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, neigh1, point, neigh1Feature, curFeature}); + } + triangleIndex++; + + triBuffer.push_back(nid2); + if(curFeature < neigh1Feature) + { + triBuffer.push_back(nid3); + triBuffer.push_back(nid4); + faceLabelBuf.push_back(curFeature); + faceLabelBuf.push_back(neigh1Feature); + ttArgsBuf.push_back({triangleIndex, neigh1, point, curFeature, neigh1Feature}); + } + else + { + triBuffer.push_back(nid4); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(neigh1Feature); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, neigh1, point, neigh1Feature, curFeature}); + } + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(neigh1Feature); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(neigh1Feature); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(neigh1Feature); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(neigh1Feature); + } + if(j == (yP - 1)) + { + MeshIndexType n1Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n2Off = (j + 1) * (xP + 1) + i; + MeshIndexType n3Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n4Off = (j + 1) * (xP + 1) + i; + assignNode(nodePlane0, n1Off, assignedNodeCount, i + 1, j + 1, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i, j + 1, k); + assignNode(nodePlane1, n3Off, assignedNodeCount, i + 1, j + 1, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + triBuffer.push_back(nid2); + triBuffer.push_back(nid4); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(-1); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(-1); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(-1); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(-1); + } + else if(curFeature != curSlice[localIdx + xP]) + { + int32 neigh2Feature = curSlice[localIdx + xP]; + MeshIndexType neigh2 = point + xP; + + MeshIndexType n1Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n2Off = (j + 1) * (xP + 1) + i; + MeshIndexType n3Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n4Off = (j + 1) * (xP + 1) + i; + assignNode(nodePlane0, n1Off, assignedNodeCount, i + 1, j + 1, k); + assignNode(nodePlane0, n2Off, assignedNodeCount, i, j + 1, k); + assignNode(nodePlane1, n3Off, assignedNodeCount, i + 1, j + 1, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane0[n1Off]; + MeshIndexType nid2 = nodePlane0[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + if(curFeature < neigh2Feature) + { + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(curFeature); + faceLabelBuf.push_back(neigh2Feature); + ttArgsBuf.push_back({triangleIndex, neigh2, point, curFeature, neigh2Feature}); + } + else + { + triBuffer.push_back(nid3); + triBuffer.push_back(nid2); + faceLabelBuf.push_back(neigh2Feature); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, neigh2, point, neigh2Feature, curFeature}); + } + triangleIndex++; + + triBuffer.push_back(nid2); + if(curFeature < neigh2Feature) + { + triBuffer.push_back(nid4); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(curFeature); + faceLabelBuf.push_back(neigh2Feature); + ttArgsBuf.push_back({triangleIndex, neigh2, point, curFeature, neigh2Feature}); + } + else + { + triBuffer.push_back(nid3); + triBuffer.push_back(nid4); + faceLabelBuf.push_back(neigh2Feature); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, neigh2, point, neigh2Feature, curFeature}); + } + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(neigh2Feature); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(neigh2Feature); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(neigh2Feature); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(neigh2Feature); + } + if(k == (zP - 1)) + { + MeshIndexType n1Off = j * (xP + 1) + (i + 1); + MeshIndexType n2Off = j * (xP + 1) + i; + MeshIndexType n3Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n4Off = (j + 1) * (xP + 1) + i; + assignNode(nodePlane1, n1Off, assignedNodeCount, i + 1, j, k + 1); + assignNode(nodePlane1, n2Off, assignedNodeCount, i, j, k + 1); + assignNode(nodePlane1, n3Off, assignedNodeCount, i + 1, j + 1, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane1[n1Off]; + MeshIndexType nid2 = nodePlane1[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + triBuffer.push_back(nid3); + triBuffer.push_back(nid2); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + triBuffer.push_back(nid4); + faceLabelBuf.push_back(-1); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, point, point, -1, curFeature}); + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(-1); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(-1); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(-1); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(-1); + } + else if(curFeature != nextSlice[localIdx]) + { + int32 neigh3Feature = nextSlice[localIdx]; + MeshIndexType neigh3 = point + sliceSize; + + MeshIndexType n1Off = j * (xP + 1) + (i + 1); + MeshIndexType n2Off = j * (xP + 1) + i; + MeshIndexType n3Off = (j + 1) * (xP + 1) + (i + 1); + MeshIndexType n4Off = (j + 1) * (xP + 1) + i; + assignNode(nodePlane1, n1Off, assignedNodeCount, i + 1, j, k + 1); + assignNode(nodePlane1, n2Off, assignedNodeCount, i, j, k + 1); + assignNode(nodePlane1, n3Off, assignedNodeCount, i + 1, j + 1, k + 1); + assignNode(nodePlane1, n4Off, assignedNodeCount, i, j + 1, k + 1); + + MeshIndexType nid1 = nodePlane1[n1Off]; + MeshIndexType nid2 = nodePlane1[n2Off]; + MeshIndexType nid3 = nodePlane1[n3Off]; + MeshIndexType nid4 = nodePlane1[n4Off]; + + triBuffer.push_back(nid1); + if(curFeature < neigh3Feature) + { + triBuffer.push_back(nid3); + triBuffer.push_back(nid2); + faceLabelBuf.push_back(curFeature); + faceLabelBuf.push_back(neigh3Feature); + ttArgsBuf.push_back({triangleIndex, neigh3, point, curFeature, neigh3Feature}); + } + else + { + triBuffer.push_back(nid2); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(neigh3Feature); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, neigh3, point, neigh3Feature, curFeature}); + } + triangleIndex++; + + triBuffer.push_back(nid2); + if(curFeature < neigh3Feature) + { + triBuffer.push_back(nid3); + triBuffer.push_back(nid4); + faceLabelBuf.push_back(curFeature); + faceLabelBuf.push_back(neigh3Feature); + ttArgsBuf.push_back({triangleIndex, neigh3, point, curFeature, neigh3Feature}); + } + else + { + triBuffer.push_back(nid4); + triBuffer.push_back(nid3); + faceLabelBuf.push_back(neigh3Feature); + faceLabelBuf.push_back(curFeature); + ttArgsBuf.push_back({triangleIndex, neigh3, point, neigh3Feature, curFeature}); + } + triangleIndex++; + + ownerLists[nid1].insert(curFeature); + ownerLists[nid1].insert(neigh3Feature); + ownerLists[nid2].insert(curFeature); + ownerLists[nid2].insert(neigh3Feature); + ownerLists[nid3].insert(curFeature); + ownerLists[nid3].insert(neigh3Feature); + ownerLists[nid4].insert(curFeature); + ownerLists[nid4].insert(neigh3Feature); + } + } + } + + // Flush buffered triangle connectivity for this z-slice + if(!triBuffer.empty()) + { + MeshIndexType sliceTriStart = triangleIndex - (triBuffer.size() / 3); + triangle.copyFromBuffer(sliceTriStart * 3, nonstd::span(triBuffer.data(), triBuffer.size())); + + // Flush buffered face labels for this z-slice + faceLabelsStore.copyFromBuffer(sliceTriStart * 2, nonstd::span(faceLabelBuf.data(), faceLabelBuf.size())); + + // Batch TupleTransfer calls with face labels embedded in the records + for(const auto& tupleTransferFunction : tupleTransferFunctions) + { + tupleTransferFunction->quickSurfaceTransferBatch(nonstd::span(ttArgsBuf.data(), ttArgsBuf.size())); + } + } + + // Rotate planes: plane1 becomes plane0 for next z-step, reinitialize plane1 + std::swap(nodePlane0, nodePlane1); + std::fill(nodePlane1.begin(), nodePlane1.end(), kMax); + + // Swap featureId buffers + std::swap(curSlice, nextSlice); + } + + // Flush all buffered vertex coordinates in one bulk write + vertex.copyFromBuffer(0, nonstd::span(vertCoordBuf.get(), nodeCount * 3)); + + // Build node types in a local buffer to avoid per-element OOC writes + auto nodeTypesBuf = std::make_unique(nodeCount); + for(usize i = 0; i < nodeCount; i++) + { + if(m_ShouldCancel) + { + return; + } + + auto& ownerList = ownerLists[i]; + + int8 nodeType = static_cast(ownerList.size()); + if(nodeType > 4) + { + nodeType = 4; + } + if(ownerList.find(-1) != ownerList.end()) + { + nodeType += 10; + } + nodeTypesBuf[i] = nodeType; + } + nodeTypesStore.copyFromBuffer(0, nonstd::span(nodeTypesBuf.get(), nodeCount)); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp new file mode 100644 index 0000000000..984ff950a6 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp @@ -0,0 +1,56 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" +#include "simplnx/Filter/IFilter.hpp" + +#include + +namespace nx::core +{ +struct QuickSurfaceMeshInputValues; + +/** + * @class QuickSurfaceMeshScanline + * @brief Out-of-core algorithm for QuickSurfaceMesh. Selected by + * DispatchAlgorithm when any input array is backed by chunked (OOC) storage. + * + * Buffers featureIds in z-slice pairs using copyIntoBuffer to avoid + * per-element virtual dispatch through AbstractDataStore::operator[]. + * The correctProblemVoxels pass uses double-buffered z-slice pairs with + * copyFromBuffer write-back. The countActiveNodesAndTriangles and + * createNodesAndTriangles passes use rolling 2-plane node buffers of + * size O((xP+1)*(yP+1)) instead of the O(volume) nodeIds array. + */ +class SIMPLNXCORE_EXPORT QuickSurfaceMeshScanline +{ +public: + using VertexStore = AbstractDataStore; + using TriStore = AbstractDataStore; + using MeshIndexType = IGeometry::MeshIndexType; + + QuickSurfaceMeshScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const QuickSurfaceMeshInputValues* inputValues); + ~QuickSurfaceMeshScanline() noexcept; + + QuickSurfaceMeshScanline(const QuickSurfaceMeshScanline&) = delete; + QuickSurfaceMeshScanline(QuickSurfaceMeshScanline&&) noexcept = delete; + QuickSurfaceMeshScanline& operator=(const QuickSurfaceMeshScanline&) = delete; + QuickSurfaceMeshScanline& operator=(QuickSurfaceMeshScanline&&) noexcept = delete; + + Result<> operator()(); + +private: + void correctProblemVoxels(); + void countActiveNodesAndTriangles(MeshIndexType& nodeCount, MeshIndexType& triangleCount, usize& numFeatures); + void createNodesAndTriangles(MeshIndexType nodeCount, MeshIndexType triangleCount, usize numFeatures); + + DataStructure& m_DataStructure; + const QuickSurfaceMeshInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp index 4706ae489e..9b7c9d7243 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp @@ -39,10 +39,21 @@ Result<> ReadHDF5Dataset::operator()() return MakeErrorResult(-21000, fmt::format("Error Reading HDF5 file: '{}'", inputFile)); } + const usize totalDatasets = datasetImportInfoList.size(); + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Reading {} dataset(s) from '{}'", totalDatasets, inputFilePath.filename().string())}); + std::map openedParentPathsMap; + usize dsIdx = 0; for(const auto& datasetImportInfo : datasetImportInfoList) { + if(m_ShouldCancel) + { + return {}; + } + std::string datasetPath = datasetImportInfo.dataSetPath; + m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Importing dataset {}/{}: '{}'", dsIdx + 1, totalDatasets, datasetPath)}); + ++dsIdx; auto datasetReader = h5FileReader.openDataset(datasetPath); std::string objectName = datasetReader.getName(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp index bab5791a57..6a96929290 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp @@ -143,9 +143,13 @@ Result<> ReadStlFile::operator()() fpos_t pos; + constexpr int32_t k_ProgressStride = 10000; for(int32_t t = 0; t < triCount; ++t) { - throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Reading {:.2f}% Complete", CalculatePercentComplete(t, triCount)); }); + if(t % k_ProgressStride == 0) + { + throttledMessenger.sendThrottledMessage([&]() { return fmt::format("Reading {:.2f}% Complete", CalculatePercentComplete(t, triCount)); }); + } if(m_ShouldCancel) { return {}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.cpp index c999d088f9..05abfa3fb5 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.cpp @@ -6,8 +6,10 @@ #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/ParallelTaskAlgorithm.hpp" +#include + #include -#include +#include using namespace nx::core; @@ -92,15 +94,17 @@ bool sliceTriangleAtZ(const std::array& verts, float32 zPlane, floa // ----------------------------------------------------------------------------- // Worker class that rasterizes a single Z-slice into a thread-local buffer, // then copies results back to the output DataArray via the parent algorithm's -// mutex-protected sendThreadSafeSliceUpdate() method. +// mutex-protected sendThreadSafeSliceUpdate() method using bulk copyFromBuffer. +// +// All input data is accessed through pre-loaded contiguous memory buffers +// (plain pointers), avoiding per-element virtual dispatch on AbstractDataStore. // ----------------------------------------------------------------------------- template class ZSliceWorker { public: ZSliceWorker(RegularGridSampleSurfaceMesh* algorithm, usize zSlice, usize xDim, usize yDim, usize numTriangles, usize numFaceLabelComps, FloatVec3 origin, FloatVec3 spacing, - const AbstractDataStore& facesRef, const AbstractDataStore& verticesRef, const AbstractDataStore& faceLabelsRef, - const std::vector& triZRanges) + const IGeometry::MeshIndexType* facesBuffer, const float32* verticesBuffer, const T* faceLabelsBuffer, const std::vector& triZRanges) : m_Algorithm(algorithm) , m_ZSlice(zSlice) , m_XDim(xDim) @@ -109,9 +113,9 @@ class ZSliceWorker , m_NumFaceLabelComps(numFaceLabelComps) , m_Origin(origin) , m_Spacing(spacing) - , m_FacesRef(facesRef) - , m_VerticesRef(verticesRef) - , m_FaceLabelsRef(faceLabelsRef) + , m_FacesBuffer(facesBuffer) + , m_VerticesBuffer(verticesBuffer) + , m_FaceLabelsBuffer(faceLabelsBuffer) , m_TriZRanges(triZRanges) { } @@ -119,7 +123,8 @@ class ZSliceWorker void operator()() const { usize cellsPerSlice = m_XDim * m_YDim; - std::vector sliceBuffer(cellsPerSlice, T{0}); + auto sliceBuffer = std::make_unique(cellsPerSlice); + std::fill(sliceBuffer.get(), sliceBuffer.get() + cellsPerSlice, T{0}); float32 zCoord = m_Origin[2] + (static_cast(m_ZSlice) + 0.5f) * m_Spacing[2]; @@ -133,12 +138,12 @@ class ZSliceWorker continue; } - usize v0Idx = m_FacesRef[t * 3]; - usize v1Idx = m_FacesRef[t * 3 + 1]; - usize v2Idx = m_FacesRef[t * 3 + 2]; - std::array verts = {Point3Df{m_VerticesRef[v0Idx * 3], m_VerticesRef[v0Idx * 3 + 1], m_VerticesRef[v0Idx * 3 + 2]}, - Point3Df{m_VerticesRef[v1Idx * 3], m_VerticesRef[v1Idx * 3 + 1], m_VerticesRef[v1Idx * 3 + 2]}, - Point3Df{m_VerticesRef[v2Idx * 3], m_VerticesRef[v2Idx * 3 + 1], m_VerticesRef[v2Idx * 3 + 2]}}; + usize v0Idx = m_FacesBuffer[t * 3]; + usize v1Idx = m_FacesBuffer[t * 3 + 1]; + usize v2Idx = m_FacesBuffer[t * 3 + 2]; + std::array verts = {Point3Df{m_VerticesBuffer[v0Idx * 3], m_VerticesBuffer[v0Idx * 3 + 1], m_VerticesBuffer[v0Idx * 3 + 2]}, + Point3Df{m_VerticesBuffer[v1Idx * 3], m_VerticesBuffer[v1Idx * 3 + 1], m_VerticesBuffer[v1Idx * 3 + 2]}, + Point3Df{m_VerticesBuffer[v2Idx * 3], m_VerticesBuffer[v2Idx * 3 + 1], m_VerticesBuffer[v2Idx * 3 + 2]}}; float32 ex1, ey1, ex2, ey2; if(sliceTriangleAtZ(verts, zCoord, ex1, ey1, ex2, ey2)) @@ -205,12 +210,12 @@ class ZSliceWorker T label0, label1; if(m_NumFaceLabelComps == 2) { - label0 = m_FaceLabelsRef[faceIdx * 2]; - label1 = m_FaceLabelsRef[faceIdx * 2 + 1]; + label0 = m_FaceLabelsBuffer[faceIdx * 2]; + label1 = m_FaceLabelsBuffer[faceIdx * 2 + 1]; } else { - label0 = m_FaceLabelsRef[faceIdx]; + label0 = m_FaceLabelsBuffer[faceIdx]; label1 = T{0}; } @@ -253,8 +258,8 @@ class ZSliceWorker } } - // ----- Copy results back to the output DataArray under mutex ----- - m_Algorithm->sendThreadSafeSliceUpdate(m_ZSlice, sliceBuffer); + // ----- Copy results back to the output DataArray under mutex via copyFromBuffer ----- + m_Algorithm->sendThreadSafeSliceUpdate(m_ZSlice, sliceBuffer.get(), cellsPerSlice); } private: @@ -266,15 +271,17 @@ class ZSliceWorker usize m_NumFaceLabelComps; FloatVec3 m_Origin; FloatVec3 m_Spacing; - const AbstractDataStore& m_FacesRef; - const AbstractDataStore& m_VerticesRef; - const AbstractDataStore& m_FaceLabelsRef; + const IGeometry::MeshIndexType* m_FacesBuffer; + const float32* m_VerticesBuffer; + const T* m_FaceLabelsBuffer; const std::vector& m_TriZRanges; }; // ----------------------------------------------------------------------------- // Functor dispatched by ExecuteDataFunctionIntType to handle the templated T. -// Sets up shared read-only data and dispatches Z-slice workers in parallel. +// Pre-loads all geometry input data into contiguous memory buffers via +// copyIntoBuffer, then dispatches Z-slice workers in parallel. Workers +// operate on plain memory pointers with no per-element virtual dispatch. // ----------------------------------------------------------------------------- struct ZSliceFunctor { @@ -283,7 +290,7 @@ struct ZSliceFunctor const ImageGeom& imageGeom, const TriangleGeom& triangleGeom, const DataPath& faceLabelsArrayPath) { // ------------------------------------------------------------------------- - // 1. Get references to input data + // 1. Get references to input data stores // ------------------------------------------------------------------------- SizeVec3 dims = imageGeom.getDimensions(); FloatVec3 origin = imageGeom.getOrigin(); @@ -293,29 +300,60 @@ struct ZSliceFunctor usize yDim = dims[1]; usize zDim = dims[2]; usize numTriangles = triangleGeom.getNumberOfFaces(); - const auto& verticesRef = triangleGeom.getVertices()->getDataStoreRef(); - const auto& facesRef = triangleGeom.getFaces()->getDataStoreRef(); + usize numVertices = triangleGeom.getNumberOfVertices(); + const auto& verticesStore = triangleGeom.getVertices()->getDataStoreRef(); + const auto& facesStore = triangleGeom.getFaces()->getDataStoreRef(); using DataArrayType = DataArray; const auto& faceLabelsArray = dataStructure.getDataRefAs(faceLabelsArrayPath); usize numFaceLabelComps = faceLabelsArray.getNumberOfComponents(); - const auto& faceLabelsRef = faceLabelsArray.getDataStoreRef(); + const auto& faceLabelsStore = faceLabelsArray.getDataStoreRef(); + + // ------------------------------------------------------------------------- + // 2. Pre-load all input geometry data into contiguous memory buffers + // via copyIntoBuffer to avoid per-element virtual dispatch on + // AbstractDataStore in the hot loops. This is proportional to triangle + // count (geometry size), not voxel count, so it's not an O(n) per-voxel + // allocation. + // ------------------------------------------------------------------------- + messageHandler({IFilter::Message::Type::Info, "Loading geometry data into memory buffers..."}); + + // Faces buffer: numTriangles * 3 elements (MeshIndexType) + usize facesCount = numTriangles * 3; + auto facesBuffer = std::make_unique(facesCount); + facesStore.copyIntoBuffer(0, nonstd::span(facesBuffer.get(), facesCount)); + + // Vertices buffer: numVertices * 3 elements (float32) + usize verticesCount = numVertices * 3; + auto verticesBuffer = std::make_unique(verticesCount); + verticesStore.copyIntoBuffer(0, nonstd::span(verticesBuffer.get(), verticesCount)); + + // Face labels buffer: numTriangles * numFaceLabelComps elements (T) + usize faceLabelsCount = numTriangles * numFaceLabelComps; + auto faceLabelsBuffer = std::make_unique(faceLabelsCount); + faceLabelsStore.copyIntoBuffer(0, nonstd::span(faceLabelsBuffer.get(), faceLabelsCount)); + + if(shouldCancel) + { + return; + } // ------------------------------------------------------------------------- - // 2. Precompute per-triangle Z-range for fast rejection + // 3. Precompute per-triangle Z-range for fast rejection using the + // pre-loaded buffers (plain memory access, no virtual dispatch) // ------------------------------------------------------------------------- messageHandler({IFilter::Message::Type::Info, "Preprocessing triangle data..."}); std::vector triZRanges(numTriangles); for(usize t = 0; t < numTriangles; t++) { - usize v0Idx = facesRef[t * 3]; - usize v1Idx = facesRef[t * 3 + 1]; - usize v2Idx = facesRef[t * 3 + 2]; + usize v0Idx = facesBuffer[t * 3]; + usize v1Idx = facesBuffer[t * 3 + 1]; + usize v2Idx = facesBuffer[t * 3 + 2]; - float32 z0 = verticesRef[v0Idx * 3 + 2]; - float32 z1 = verticesRef[v1Idx * 3 + 2]; - float32 z2 = verticesRef[v2Idx * 3 + 2]; + float32 z0 = verticesBuffer[v0Idx * 3 + 2]; + float32 z1 = verticesBuffer[v1Idx * 3 + 2]; + float32 z2 = verticesBuffer[v2Idx * 3 + 2]; triZRanges[t].zMin = std::min({z0, z1, z2}); triZRanges[t].zMax = std::max({z0, z1, z2}); @@ -327,7 +365,8 @@ struct ZSliceFunctor } // ------------------------------------------------------------------------- - // 3. Dispatch Z-slices in parallel using ParallelTaskAlgorithm + // 4. Dispatch Z-slices in parallel using ParallelTaskAlgorithm. + // Workers read from the pre-loaded buffers (plain pointers). // ------------------------------------------------------------------------- messageHandler({IFilter::Message::Type::Info, fmt::format("Sampling surface mesh using scanline rasterization ({} Z-slices)...", zDim)}); @@ -339,7 +378,7 @@ struct ZSliceFunctor break; } - taskAlgorithm.execute(ZSliceWorker(algorithm, z, xDim, yDim, numTriangles, numFaceLabelComps, origin, spacing, facesRef, verticesRef, faceLabelsRef, triZRanges)); + taskAlgorithm.execute(ZSliceWorker(algorithm, z, xDim, yDim, numTriangles, numFaceLabelComps, origin, spacing, facesBuffer.get(), verticesBuffer.get(), faceLabelsBuffer.get(), triZRanges)); } taskAlgorithm.wait(); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.hpp index 8deb064011..2f3e817d07 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RegularGridSampleSurfaceMesh.hpp @@ -33,7 +33,12 @@ struct SIMPLNXCORE_EXPORT RegularGridSampleSurfaceMeshInputValues * * Z-slices are processed in parallel. Each worker thread rasterizes into * a thread-local buffer and then copies results back to the output DataArray - * under a mutex, ensuring thread safety with out-of-core DataStore implementations. + * under a mutex via copyFromBuffer, ensuring thread safety and efficient + * bulk I/O with out-of-core DataStore implementations. + * + * All input geometry data (faces, vertices, face labels) is pre-loaded into + * contiguous memory buffers via copyIntoBuffer at algorithm start, so worker + * threads operate on plain memory arrays with no virtual dispatch per element. */ class SIMPLNXCORE_EXPORT RegularGridSampleSurfaceMesh { @@ -50,20 +55,20 @@ class SIMPLNXCORE_EXPORT RegularGridSampleSurfaceMesh /** * @brief Thread-safe method to copy a completed Z-slice buffer into the - * output DataArray. Called by worker threads after rasterizing a slice. + * output DataArray using bulk copyFromBuffer. Called by worker threads + * after rasterizing a slice. + * @tparam T The element type of the feature IDs array * @param zSlice The Z-slice index - * @param sliceBuffer The thread-local buffer containing rasterized feature IDs + * @param sliceData Raw pointer to the thread-local buffer containing rasterized feature IDs + * @param count Number of elements in the slice buffer */ template - void sendThreadSafeSliceUpdate(usize zSlice, const std::vector& sliceBuffer) + void sendThreadSafeSliceUpdate(usize zSlice, const T* sliceData, usize count) { std::lock_guard lock(m_Mutex); auto& featureIdsRef = m_DataStructure.getDataRefAs>(m_InputValues->FeatureIdsArrayPath).getDataStoreRef(); usize offset = zSlice * m_CellsPerSlice; - for(usize i = 0; i < m_CellsPerSlice; i++) - { - featureIdsRef[offset + i] = sliceBuffer[i]; - } + featureIdsRef.copyFromBuffer(offset, nonstd::span(sliceData, count)); } private: diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp index e7fb389196..51a7b9c61f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp @@ -4,6 +4,7 @@ #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Utilities/FilterUtilities.hpp" #include "simplnx/Utilities/NeighborUtilities.hpp" +#include "simplnx/Utilities/SliceBufferedTransfer.hpp" using namespace nx::core; @@ -82,23 +83,22 @@ class GreaterThanComparison : public IComparisonFunctor struct ExecuteTemplate { template - void CompareValues(std::shared_ptr>& comparator, const AbstractDataStore& inputArray, int64 neighbor, float thresholdValue, float32& best, - std::vector& bestNeighbor, size_t i) const + void CompareValues(std::shared_ptr>& comparator, T neighborValue, float32 ThresholdValue, float32& best, std::vector& bestNeighbor, usize i, int64 neighborPoint) const { - if(comparator->compare1(inputArray[neighbor], thresholdValue) && comparator->compare2(inputArray[neighbor], best)) + if(comparator->compare1(neighborValue, ThresholdValue) && comparator->compare2(neighborValue, best)) { - best = inputArray[neighbor]; - bestNeighbor[i] = neighbor; + best = neighborValue; + bestNeighbor[i] = neighborPoint; } } template - void operator()(const ImageGeom& imageGeom, IDataArray* inputIDataArray, int32 comparisonAlgorithm, float thresholdValue, bool loopUntilDone, const std::atomic_bool& shouldCancel, + void operator()(const ImageGeom& imageGeom, IDataArray* inputIDataArray, int32 comparisonAlgorithm, float32 ThresholdValue, bool loopUntilDone, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& messageHandler) { const auto& inputStore = inputIDataArray->template getIDataStoreRefAs>(); - const size_t totalPoints = inputStore.getNumberOfTuples(); + const usize totalPoints = inputStore.getNumberOfTuples(); Vec3 udims = imageGeom.getDimensions(); std::array dims = { @@ -107,16 +107,10 @@ struct ExecuteTemplate static_cast(udims[2]), }; - // bool good = true; - int64 neighbor = 0; - int64 column = 0; - int64 row = 0; - int64 plane = 0; - std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); - std::vector bestNeighbor(totalPoints, -1); + std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - size_t count = 0; + usize count = 0; bool keepGoing = true; std::shared_ptr> comparator = std::make_shared>(); @@ -127,6 +121,30 @@ struct ExecuteTemplate const AttributeMatrix* attrMatrix = imageGeom.getCellData(); + // Z-slice buffering: maintain rolling window of 3 adjacent Z-slices for input array + // to avoid random OOC chunk access during neighbor lookups. + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + const usize dimZ = static_cast(dims[2]); + + // Per-slice best neighbor marks (replaces O(totalPoints) bestNeighbor array) + std::vector sliceBestNeighbor(sliceSize, -1); + + // Rolling window: slot 0 = z-1, slot 1 = z (current), slot 2 = z+1 + // Use unique_ptr instead of std::vector to avoid std::vector bit-packing + std::array, 3> inputSlices; + for(auto& is : inputSlices) + { + is = std::make_unique(sliceSize); + } + + auto readInputSlice = [&](int64 z, usize slot) { + const usize zOffset = static_cast(z) * sliceSize; + inputStore.copyIntoBuffer(zOffset, nonstd::span(inputSlices[slot].get(), sliceSize)); + }; + + // Face neighbor ordering: 0=-Z, 1=-Y, 2=-X, 3=+X, 4=+Y, 5=+Z + constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + while(keepGoing) { keepGoing = false; @@ -136,57 +154,88 @@ struct ExecuteTemplate break; } - auto progIncrement = static_cast(totalPoints / 50); + // Initialize rolling window: load z=0 into slot 1, z=1 into slot 2 + readInputSlice(0, 1); + if(dims[2] > 1) + { + readInputSlice(1, 2); + } + + auto progIncrement = static_cast(totalPoints / 50); int64 prog = 1; int64 progressInt = 0; - for(size_t voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) + + for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - if(comparator->compare(inputStore[voxelIndex], thresholdValue)) + // Advance rolling window for z > 0 + if(zIdx > 0) { - column = voxelIndex % dims[0]; - row = (voxelIndex / dims[0]) % dims[1]; - plane = voxelIndex / (dims[0] * dims[1]); - count++; - float32 best = inputStore[voxelIndex]; - - neighbor = static_cast(voxelIndex) + neighborVoxelIndexOffsets[0]; - if(plane != 0) - { - CompareValues(comparator, inputStore, neighbor, thresholdValue, best, bestNeighbor, voxelIndex); - } - neighbor = static_cast(voxelIndex) + neighborVoxelIndexOffsets[1]; - if(row != 0) - { - CompareValues(comparator, inputStore, neighbor, thresholdValue, best, bestNeighbor, voxelIndex); - } - neighbor = static_cast(voxelIndex) + neighborVoxelIndexOffsets[2]; - if(column != 0) - { - CompareValues(comparator, inputStore, neighbor, thresholdValue, best, bestNeighbor, voxelIndex); - } - neighbor = static_cast(voxelIndex) + neighborVoxelIndexOffsets[3]; - if(column != (dims[0] - 1)) + std::swap(inputSlices[0], inputSlices[1]); + std::swap(inputSlices[1], inputSlices[2]); + if(zIdx + 1 < dims[2]) { - CompareValues(comparator, inputStore, neighbor, thresholdValue, best, bestNeighbor, voxelIndex); + readInputSlice(zIdx + 1, 2); } - neighbor = static_cast(voxelIndex) + neighborVoxelIndexOffsets[4]; - if(row != (dims[1] - 1)) - { - CompareValues(comparator, inputStore, neighbor, thresholdValue, best, bestNeighbor, voxelIndex); - } - neighbor = static_cast(voxelIndex) + neighborVoxelIndexOffsets[5]; - if(plane != (dims[2] - 1)) + } + + for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) + { + for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { - CompareValues(comparator, inputStore, neighbor, thresholdValue, best, bestNeighbor, voxelIndex); + const int64 voxelIndex = zIdx * static_cast(sliceSize) + yIdx * dims[0] + xIdx; + const usize inSlice = static_cast(yIdx * dims[0] + xIdx); + + if(comparator->compare(inputSlices[1][inSlice], ThresholdValue)) + { + count++; + float32 best = inputSlices[1][inSlice]; + + std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + + const std::array neighborInSlice = { + inSlice, // -Z + static_cast((yIdx - 1) * dims[0] + xIdx), // -Y + static_cast(yIdx * dims[0] + (xIdx - 1)), // -X + static_cast(yIdx * dims[0] + (xIdx + 1)), // +X + static_cast((yIdx + 1) * dims[0] + xIdx), // +Y + inSlice // +Z + }; + + for(const auto& faceIndex : faceNeighborInternalIdx) + { + if(!isValidFaceNeighbor[faceIndex]) + { + continue; + } + + const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + const T neighborValue = inputSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; + CompareValues(comparator, neighborValue, ThresholdValue, best, sliceBestNeighbor, inSlice, neighborPoint); + } + } + if(voxelIndex > prog) + { + progressInt = static_cast((static_cast(voxelIndex) / totalPoints) * 100.0f); + const std::string progressMessage = fmt::format("Processing Loop({}) Progress: {}% Complete", count, progressInt); + messageHandler(IFilter::ProgressMessage{IFilter::Message::Type::Progress, progressMessage, static_cast(progressInt)}); + prog += progIncrement; + } } } - if(voxelIndex > prog) + + // Transfer this Z-slice immediately (bestNeighbor only marks current voxel, not cross-slice) + for(const auto& [dataId, dataObject] : *attrMatrix) { - progressInt = static_cast(((float)voxelIndex / totalPoints) * 100.0f); - const std::string progressMessage = fmt::format("Processing Loop({}) Progress: {}% Complete", count, progressInt); - messageHandler(IFilter::ProgressMessage{IFilter::Message::Type::Progress, progressMessage, static_cast(progressInt)}); - prog += progIncrement; + auto* dataArrayPtr = dynamic_cast(dataObject.get()); + if(dataArrayPtr == nullptr) + { + continue; + } + SliceBufferedTransferOneZ(*dataArrayPtr, sliceBestNeighbor, sliceSize, static_cast(zIdx), dimZ); } + + // Clear per-slice marks for next Z + std::fill(sliceBestNeighbor.begin(), sliceBestNeighbor.end(), -1); } if(shouldCancel) @@ -194,29 +243,6 @@ struct ExecuteTemplate break; } - progIncrement = static_cast(totalPoints / 50); - prog = 1; - progressInt = 0; - for(int64 voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) - { - if(voxelIndex > prog) - { - progressInt = static_cast(((float)voxelIndex / totalPoints) * 100.0f); - const std::string progressMessage = fmt::format("Transferring Loop({}) Progress: {}% Complete", count, progressInt); - messageHandler(IFilter::ProgressMessage{IFilter::Message::Type::Progress, progressMessage, static_cast(progressInt)}); - prog = prog + progIncrement; - } - - neighbor = bestNeighbor[voxelIndex]; - if(neighbor != -1) - { - for(const auto& [dataId, dataObject] : *attrMatrix) - { - auto& dataArray = dynamic_cast(*dataObject); - dataArray.copyTuple(neighbor, voxelIndex); - } - } - } if(loopUntilDone && count > 0) { keepGoing = true; @@ -241,7 +267,7 @@ ReplaceElementAttributesWithNeighborValues::ReplaceElementAttributesWithNeighbor ReplaceElementAttributesWithNeighborValues::~ReplaceElementAttributesWithNeighborValues() noexcept = default; // ----------------------------------------------------------------------------- -const std::atomic_bool& ReplaceElementAttributesWithNeighborValues::getCancel() +const std::atomic_bool& ReplaceElementAttributesWithNeighborValues::getCancel() const { return m_ShouldCancel; } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp index e32fc157b5..beb606207e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp @@ -19,6 +19,10 @@ inline constexpr StringLiteral k_GreaterThan = "> [Greater Than]"; inline const ChoicesParameter::Choices k_OperationChoices = {k_LessThan, k_GreaterThan}; } // namespace detail +/** + * @struct ReplaceElementAttributesWithNeighborValuesInputValues + * @brief Holds all user-supplied parameters for the ReplaceElementAttributesWithNeighborValues algorithm. + */ struct SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValuesInputValues { float32 MinConfidence; @@ -29,13 +33,25 @@ struct SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValuesInputValues }; /** - * @class + * @class ReplaceElementAttributesWithNeighborValues + * @brief Replaces voxel data with the best face-neighbor value based on a threshold comparison. */ class SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValues { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays + * @param mesgHandler Handler for sending progress messages to the UI + * @param shouldCancel Atomic flag checked between iterations to support cancellation + * @param inputValues User-supplied parameters controlling the algorithm behavior + */ ReplaceElementAttributesWithNeighborValues(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReplaceElementAttributesWithNeighborValuesInputValues* inputValues); + + /** + * @brief Default destructor. + */ ~ReplaceElementAttributesWithNeighborValues() noexcept; ReplaceElementAttributesWithNeighborValues(const ReplaceElementAttributesWithNeighborValues&) = delete; @@ -43,9 +59,13 @@ class SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValues ReplaceElementAttributesWithNeighborValues& operator=(const ReplaceElementAttributesWithNeighborValues&) = delete; ReplaceElementAttributesWithNeighborValues& operator=(ReplaceElementAttributesWithNeighborValues&&) noexcept = delete; + /** + * @brief Executes the replace element attributes with neighbor values algorithm. + * @return Result<> indicating success or any errors encountered during execution + */ Result<> operator()(); - const std::atomic_bool& getCancel(); + const std::atomic_bool& getCancel() const; private: DataStructure& m_DataStructure; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp index 5a0edad7ca..23dd4b37aa 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp @@ -8,8 +8,17 @@ #include "simplnx/Utilities/ParallelTaskAlgorithm.hpp" #include "simplnx/Utilities/TimeUtilities.hpp" +#include + +#include + using namespace nx::core; +namespace +{ +constexpr usize k_ChunkTuples = 65536; +} // namespace + namespace { @@ -124,6 +133,7 @@ Result<> RequireMinimumSizeFeatures::operator()() Error errorReturn = {0, ""}; std::vector activeObjects = removeSmallFeatures(featureIdsStoreRef, featureNumCellsStoreRef, featurePhases, m_InputValues->PhaseNumber, m_InputValues->ApplySinglePhase, m_InputValues->MinAllowedFeaturesSize, errorReturn); + if(errorReturn.code < 0) { return {nonstd::make_unexpected(std::vector{errorReturn})}; @@ -135,6 +145,7 @@ Result<> RequireMinimumSizeFeatures::operator()() auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometryPath); assignBadVoxels(imageGeom.getDimensions(), featureNumCellsStoreRef); + if(m_ShouldCancel) { return {}; @@ -172,28 +183,42 @@ void RequireMinimumSizeFeatures::assignBadVoxels(SizeVec3 dimensions, const Int3 static_cast(dimensions[2]), }; - std::vector neighborsVoxelIndex(totalPoints * featureIds.getNumberOfComponents(), -1); + // Track which voxels need data copied from a neighbor + std::vector neighborsVoxelIndex(totalPoints, -1); - // int32 good = 1; int64 neighborVoxelIdx = 0; - // These are the offsets that are applied to a voxel index to get to a specific neighbor voxel std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); usize counter = 1; - int64 count = 0; int64 kstride = 0; int64 jstride = 0; - // `voteCounter` serves as a vote counter array for determining which feature ID should - // be assigned to "bad" voxels (those with featureId < 0 after small features - // were removed). The array indexing is by feature ID. The largest value that could - // be saved is 6 since there are only 6 face neighbors. std::vector voteCounter(featureNumCellsStoreRef.getNumberOfTuples(), 0); + // Chunked scan: read featureIds in chunks for the voting scan + const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); + while(counter != 0) { counter = 0; + + // Scan phase: read featureIds in z-slice chunks for voting + // Use 3-slice rolling buffer so all 6 face neighbors are accessible + std::vector slabBuf(3 * sliceSize, 0); + int32* prevSlice = slabBuf.data(); + int32* curSlice = slabBuf.data() + sliceSize; + int32* nextSlice = slabBuf.data() + 2 * sliceSize; + + featureIds.copyIntoBuffer(0, nonstd::span(curSlice, sliceSize)); + if(dims[2] > 1) + { + featureIds.copyIntoBuffer(sliceSize, nonstd::span(nextSlice, sliceSize)); + } + + // Collect indices of voxels that get assigned this iteration + std::vector changedVoxels; + for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { if(m_ShouldCancel) @@ -206,13 +231,13 @@ void RequireMinimumSizeFeatures::assignBadVoxels(SizeVec3 dimensions, const Int3 jstride = dims[0] * yIdx; for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { - count = kstride + jstride + xIdx; - int32 currentFeatureId = featureIds.getValue(count); + const int64 globalIdx = kstride + jstride + xIdx; + const int64 localIdx = yIdx * dims[0] + xIdx; + int32 currentFeatureId = curSlice[localIdx]; if(currentFeatureId < 0) { counter++; uint8 maxVoteCount = 0; - // Loop over the 6 face neighbors of the voxel std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); for(const auto& faceIndex : faceNeighborInternalIdx) { @@ -221,8 +246,24 @@ void RequireMinimumSizeFeatures::assignBadVoxels(SizeVec3 dimensions, const Int3 continue; } - neighborVoxelIdx = count + neighborVoxelIndexOffsets[faceIndex]; - int32 neighborFeatureId = featureIds.getValue(neighborVoxelIdx); + neighborVoxelIdx = globalIdx + neighborVoxelIndexOffsets[faceIndex]; + + // Read neighbor from slab buffer + int32 neighborFeatureId = 0; + const int64 neighborOffset = neighborVoxelIndexOffsets[faceIndex]; + if(neighborOffset == -dims[0] * dims[1]) + { + neighborFeatureId = prevSlice[localIdx]; + } + else if(neighborOffset == dims[0] * dims[1]) + { + neighborFeatureId = nextSlice[localIdx]; + } + else + { + neighborFeatureId = curSlice[localIdx + neighborOffset]; + } + if(neighborFeatureId >= 0) { voteCounter[neighborFeatureId]++; @@ -230,41 +271,62 @@ void RequireMinimumSizeFeatures::assignBadVoxels(SizeVec3 dimensions, const Int3 if(currentVoteCount > maxVoteCount) { maxVoteCount = currentVoteCount; - neighborsVoxelIndex[count] = neighborVoxelIdx; + neighborsVoxelIndex[globalIdx] = neighborVoxelIdx; } } } - // Reset the VoteCounter back to Zero... + if(neighborsVoxelIndex[globalIdx] >= 0) + { + changedVoxels.push_back(static_cast(globalIdx)); + } + std::fill(voteCounter.begin(), voteCounter.end(), 0); } } } + + // Slide the slab window + std::swap(prevSlice, curSlice); + std::swap(curSlice, nextSlice); + if(zIdx + 2 < dims[2]) + { + featureIds.copyIntoBuffer(static_cast(zIdx + 2) * sliceSize, nonstd::span(nextSlice, sliceSize)); + } + } + + // Skip transfer entirely if no voxels were assigned + if(changedVoxels.empty()) + { + break; } - messageHelper.sendMessage(fmt::format("Remaining voxels: {} - Updating Data Arrays... ", counter)); + messageHelper.sendMessage(fmt::format("Remaining voxels: {} - Updating {} changed voxels... ", counter, changedVoxels.size())); - // Build up a list of the DataArrays that we are going to operate on. + // Transfer phase: only copy tuples for voxels that actually changed const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsPath, {}); - ParallelTaskAlgorithm taskRunner; - taskRunner.setParallelizationEnabled(true); for(const auto& voxelArray : voxelArrays) { - // We need to skip updating the FeatureIds until all the other arrays are updated - // since we actually depend on the feature Ids values. - if(voxelArray->getName() == m_InputValues->FeatureIdsPath.getTargetName()) + if(m_ShouldCancel) + { + return; + } + for(const usize voxelIndex : changedVoxels) { - continue; + int64 neighborIdx = neighborsVoxelIndex[voxelIndex]; + if(neighborIdx >= 0 && featureIds.getValue(neighborIdx) >= 0) + { + voxelArray->copyTuple(neighborIdx, voxelIndex); + } } + } - taskRunner.execute(RequireMinimumSizeFeaturesTransferDataImpl(this, totalPoints, featureIds, neighborsVoxelIndex, voxelArray, messageHelper, m_ShouldCancel)); + // Reset neighborsVoxelIndex for changed voxels + for(const usize voxelIndex : changedVoxels) + { + neighborsVoxelIndex[voxelIndex] = -1; } - taskRunner.wait(); // This will spill over if the number of DataArrays to process does not divide evenly by the number of threads. - // Now update the feature Ids - auto featureIDataArray = m_DataStructure.getSharedDataAs(m_InputValues->FeatureIdsPath); - taskRunner.setParallelizationEnabled(false); // Do this to make the next call synchronous - taskRunner.execute(RequireMinimumSizeFeaturesTransferDataImpl(this, totalPoints, featureIds, neighborsVoxelIndex, featureIDataArray, messageHelper, m_ShouldCancel)); } } @@ -279,7 +341,6 @@ std::vector RequireMinimumSizeFeatures::removeSmallFeatures(Int32AbstractD usize totalPoints = featureIdsStoreRef.getNumberOfTuples(); bool good = false; - int32 gnum; usize totalFeatures = featureNumCellsStoreRef.getNumberOfTuples(); @@ -319,12 +380,29 @@ std::vector RequireMinimumSizeFeatures::removeSmallFeatures(Int32AbstractD errorReturn = Error{-1, "The minimum size is larger than the largest Feature. All Features would be removed"}; return activeObjects; } - for(usize i = 0; i < totalPoints; i++) + + auto featureIdBuf = std::make_unique(k_ChunkTuples); + for(usize offset = 0; offset < totalPoints; offset += k_ChunkTuples) { - gnum = featureIdsStoreRef.getValue(i); - if(!activeObjects[gnum]) + if(m_ShouldCancel) + { + return {}; + } + const usize count = std::min(k_ChunkTuples, totalPoints - offset); + featureIdsStoreRef.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), count)); + + bool modified = false; + for(usize i = 0; i < count; i++) + { + if(!activeObjects[featureIdBuf[i]]) + { + featureIdBuf[i] = -1; + modified = true; + } + } + if(modified) { - featureIdsStoreRef.setValue(i, -1); + featureIdsStoreRef.copyFromBuffer(offset, nonstd::span(featureIdBuf.get(), count)); } } return activeObjects; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.cpp index 51805d1765..5450a0d2d5 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.cpp @@ -1,10 +1,13 @@ #include "ScalarSegmentFeatures.hpp" +#include #include #include "simplnx/DataStructure/DataStore.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" #include "simplnx/Filter/Actions/CreateArrayAction.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" using namespace nx::core; @@ -54,6 +57,15 @@ class TSpecificCompareFunctorBool : public SegmentFeatures::CompareFunctor return false; } + bool compare(int64 index, int64 neighIndex) override + { + if(index >= m_Length || neighIndex >= m_Length) + { + return false; + } + return (*m_Data)[neighIndex] == (*m_Data)[index]; + } + private: int64 m_Length = 0; // Length of the Data Array AbstractDataStore* m_FeatureIdsArray = nullptr; // The Feature Ids @@ -109,12 +121,48 @@ class TSpecificCompareFunctor : public SegmentFeatures::CompareFunctor return false; } + bool compare(int64 index, int64 neighIndex) override + { + if(index >= m_Length || neighIndex >= m_Length) + { + return false; + } + + if(m_Data[index] >= m_Data[neighIndex]) + { + return (m_Data[index] - m_Data[neighIndex]) <= m_Tolerance; + } + return (m_Data[neighIndex] - m_Data[index]) <= m_Tolerance; + } + private: int64 m_Length = 0; // Length of the Data Array T m_Tolerance = static_cast(0); // The tolerance of the comparison AbstractDataStore* m_FeatureIdsArray = nullptr; // The Feature Ids DataStoreType& m_Data; // The data that is being compared }; + +/** + * @brief Functor for type-dispatched filling of a scalar slice buffer. + * Bulk-reads an entire slice via copyIntoBuffer() into a local typed buffer, + * then converts to float64 for uniform comparison. Uses std::make_unique + * instead of std::vector to avoid std::vector specialization issues. + */ +struct FillScalarSliceBufferFunctor +{ + template + void operator()(IDataArray* dataArray, int64 baseIndex, usize sliceSize, std::vector& buffer, usize bufferOffset) + { + auto& store = dataArray->template getIDataStoreRefAs>(); + // Bulk-read the entire slice into a local typed buffer, then convert to float64 + auto tempBuffer = std::make_unique(sliceSize); + store.copyIntoBuffer(static_cast(baseIndex), nonstd::span(tempBuffer.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + buffer[bufferOffset + i] = static_cast(tempBuffer[i]); + } + } +}; } // namespace ScalarSegmentFeatures::ScalarSegmentFeatures(DataStructure& dataStructure, ScalarSegmentFeaturesInputValues* inputValues, const std::atomic_bool& shouldCancel, @@ -127,6 +175,39 @@ ScalarSegmentFeatures::ScalarSegmentFeatures(DataStructure& dataStructure, Scala ScalarSegmentFeatures::~ScalarSegmentFeatures() noexcept = default; +// ----------------------------------------------------------------------------- +// Segments an image/rectilinear grid into features (regions) by flood-filling +// contiguous voxels whose scalar values differ by no more than a user-specified +// tolerance. This is a general-purpose segmentation: it works on any single- +// component scalar array (int8 through float64, plus boolean), unlike the +// orientation-based EBSD and CAxis segment filters. +// +// Comparator setup: +// A type-dispatched CompareFunctor is instantiated via a switch on the input +// array's DataType. Each TSpecificCompareFunctor stores the tolerance cast +// to the native type and performs |a - b| <= tolerance using unsigned-safe +// subtraction. Boolean arrays use a dedicated TSpecificCompareFunctorBool +// that checks for exact equality (no tolerance concept). If the input array +// has more than one component, a default CompareFunctor that always returns +// false is used, effectively preventing any grouping. +// +// Algorithm dispatch: +// - In-core data -> execute() : classic depth-first-search (DFS) flood fill +// - Out-of-core -> executeCCL() : connected-component labeling that streams +// data slice-by-slice to limit memory usage +// The choice is made by checking IsOutOfCore() on the FeatureIds array (i.e., +// whether the backing DataStore lives on disk) or if ForceOocAlgorithm() is +// set (used for testing). +// +// Post-processing after either algorithm: +// 1. Validate that at least one feature was found (error if not). +// 2. Resize the Feature AttributeMatrix to (m_FoundFeatures + 1) tuples so +// that all per-feature arrays (Active, etc.) have the correct size. +// Index 0 is reserved as an invalid/background feature. +// 3. Initialize the Active array: fill with 1 (active), then set index 0 +// to 0 to mark it as the reserved background slot. +// 4. Optionally randomize FeatureIds so that spatially adjacent features get +// non-sequential IDs, improving visual contrast in color-mapped renders. // ----------------------------------------------------------------------------- Result<> ScalarSegmentFeatures::operator()() { @@ -150,7 +231,8 @@ Result<> ScalarSegmentFeatures::operator()() m_FeatureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); auto* inputDataArray = m_DataStructure.getDataAs(m_InputValues->InputDataPath); - size_t inDataPoints = inputDataArray->getNumberOfTuples(); + m_InputDataArray = inputDataArray; + usize inDataPoints = inputDataArray->getNumberOfTuples(); nx::core::DataType dataType = inputDataArray->getDataType(); auto featureIds = m_FeatureIdsArray->getDataStore(); @@ -209,8 +291,21 @@ Result<> ScalarSegmentFeatures::operator()() m_CompareFunctor = std::make_shared(); // The default CompareFunctor which ALWAYS returns false for the comparison } - // Run the segmentation algorithm - execute(gridGeom); + // Dispatch between DFS (in-core) and CCL (OOC) algorithms + if(IsOutOfCore(*m_FeatureIdsArray) || ForceOocAlgorithm()) + { + SizeVec3 udims = gridGeom->getDimensions(); + allocateSliceBuffers(static_cast(udims[0]), static_cast(udims[1])); + + auto& featureIdsStore = m_FeatureIdsArray->getDataStoreRef(); + executeCCL(gridGeom, featureIdsStore); + + deallocateSliceBuffers(); + } + else + { + execute(gridGeom); + } // Sanity check the result. if(this->m_FoundFeatures < 1) { @@ -223,8 +318,9 @@ Result<> ScalarSegmentFeatures::operator()() cellFeaturesAM.resizeTuples(tDims); // This will resize the active array // make sure all values are initialized and "re-reserve" index 0 - auto& activeStore = m_DataStructure.getDataAs(m_InputValues->ActiveArrayPath)->getDataStoreRef(); - activeStore[0] = 0; + auto* activeArray = m_DataStructure.getDataAs(m_InputValues->ActiveArrayPath); + activeArray->getDataStore()->fill(1); + (*activeArray)[0] = 0; // Randomize the feature Ids for purely visual clarify. Having random Feature Ids // allows users visualizing the data to better discern each grain otherwise the coloring @@ -238,7 +334,25 @@ Result<> ScalarSegmentFeatures::operator()() } // ----------------------------------------------------------------------------- -int64_t ScalarSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const +// Finds the next unassigned voxel that can serve as the seed for a new feature. +// The scan is a simple linear walk starting from `nextSeed`, which is the index +// immediately after the last seed found. This avoids rescanning already-assigned +// voxels at the beginning of the array. +// +// A voxel is eligible to become a seed when both conditions are met: +// 1. featureId == 0 : the voxel has not yet been assigned to any feature. +// 2. Passes the mask: if masking is enabled, the voxel must be flagged as +// "good" (e.g., not a bad scan point). +// +// Note: Unlike EBSD and CAxis segmentation, there is no phase > 0 requirement +// because scalar segmentation is phase-agnostic -- it operates on arbitrary +// numeric data that has no concept of crystallographic phase. +// +// When a valid seed is found, its featureId is immediately set to `gnum` +// (the new feature number) so that subsequent calls will skip it. +// Returns the linear index of the seed, or -1 if no more seeds exist. +// ----------------------------------------------------------------------------- +int64 ScalarSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const { nx::core::DataArray::store_type* featureIds = m_FeatureIdsArray->getDataStore(); usize totalPoints = featureIds->getNumberOfTuples(); @@ -271,6 +385,19 @@ int64_t ScalarSegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const return seed; } +// ----------------------------------------------------------------------------- +// Determines whether a neighboring voxel should be merged into the current +// feature during the DFS flood fill (execute() path). This is NOT used by +// the CCL path, which calls areNeighborsSimilar() instead. +// +// The method checks two conditions before grouping: +// 1. The neighbor's featureId must be 0 (unassigned). +// 2. The neighbor must pass the mask (if masking is enabled). +// +// If both conditions pass, the type-dispatched CompareFunctor is invoked. +// The functor checks whether |scalar[reference] - scalar[neighbor]| <= tolerance +// (for numeric types) or exact equality (for booleans). As a side effect, the +// functor assigns featureId = gnum to the neighbor if the comparison succeeds. // ----------------------------------------------------------------------------- bool ScalarSegmentFeatures::determineGrouping(int64 referencepoint, int64 neighborpoint, int32 gnum) const { @@ -283,3 +410,261 @@ bool ScalarSegmentFeatures::determineGrouping(int64 referencepoint, int64 neighb return false; } + +// ----------------------------------------------------------------------------- +// Checks whether a single voxel is eligible for segmentation (used by the CCL +// path in executeCCL()). For scalar segmentation, validity only depends on the +// mask -- there is no phase check because scalar data is phase-agnostic. +// +// Slice buffer fast path: +// When m_UseSliceBuffers is true (OOC mode), the method first checks whether +// the voxel's Z-slice is currently loaded in the rolling 2-slot buffer. The +// slot is determined by (iz % 2). If the voxel's slice matches the buffered +// slice index, the mask value is read directly from the in-memory m_MaskBuffer, +// avoiding an on-disk I/O round-trip. +// +// OOC fallback: +// If slice buffers are not active, or if the voxel's slice is not currently +// buffered (which can happen during Phase 1b of CCL when periodic boundary +// merging accesses non-adjacent slices), the method falls back to direct +// MaskCompare access, which may trigger on-disk I/O for out-of-core data. +// ----------------------------------------------------------------------------- +bool ScalarSegmentFeatures::isValidVoxel(int64 point) const +{ + if(m_UseSliceBuffers) + { + const int64 iz = point / m_BufSliceSize; + const int slot = static_cast(iz % 2); + if(m_BufferedSliceZ[slot] == iz) + { + const usize off = static_cast(slot) * static_cast(m_BufSliceSize) + static_cast(point - iz * m_BufSliceSize); + if(m_InputValues->UseMask && m_MaskBuffer[off] == 0) + { + return false; + } + return true; + } + } + + // Fallback: direct OOC access + if(m_InputValues->UseMask && !m_GoodVoxels->isTrue(point)) + { + return false; + } + return true; +} + +// ----------------------------------------------------------------------------- +// Determines whether two neighboring voxels have sufficiently similar scalar +// values to belong to the same feature. Used exclusively by the CCL path +// (executeCCL()), whereas the DFS path uses determineGrouping() instead. +// +// Slice buffer fast path: +// When both voxels' Z-slices are present in the rolling 2-slot buffer, all +// data is read from the in-memory buffers (m_ScalarBuffer, m_MaskBuffer). +// The buffer offset for each point is computed as: +// slot * sliceSize + (point - iz * sliceSize) +// The method then: +// 1. Checks point2's mask validity. +// 2. Reads both scalar values from m_ScalarBuffer as float64. +// 3. Computes |val1 - val2| and returns true if <= ScalarTolerance. +// All scalar types are stored as float64 in the buffer so that a single +// comparison path works regardless of the original data type. The tolerance +// is also cast to float64 for the comparison. +// +// OOC fallback: +// If either voxel's slice is not buffered (e.g., during Phase 1b periodic +// merge), falls back to direct DataStore access: validates point2 via +// isValidVoxel(), then delegates to m_CompareFunctor->compare() which +// reads from the original typed array on disk. +// ----------------------------------------------------------------------------- +bool ScalarSegmentFeatures::areNeighborsSimilar(int64 point1, int64 point2) const +{ + if(m_UseSliceBuffers) + { + const int64 iz1 = point1 / m_BufSliceSize; + const int slot1 = static_cast(iz1 % 2); + const int64 iz2 = point2 / m_BufSliceSize; + const int slot2 = static_cast(iz2 % 2); + + if(m_BufferedSliceZ[slot1] == iz1 && m_BufferedSliceZ[slot2] == iz2) + { + const usize sliceSize = static_cast(m_BufSliceSize); + const usize off1 = static_cast(slot1) * sliceSize + static_cast(point1 - iz1 * m_BufSliceSize); + const usize off2 = static_cast(slot2) * sliceSize + static_cast(point2 - iz2 * m_BufSliceSize); + + // Check point2 validity + if(m_InputValues->UseMask && m_MaskBuffer[off2] == 0) + { + return false; + } + + // Compare scalar values from the pre-loaded buffer + float64 val1 = m_ScalarBuffer[off1]; + float64 val2 = m_ScalarBuffer[off2]; + float64 diff = val1 >= val2 ? (val1 - val2) : (val2 - val1); + return diff <= static_cast(m_InputValues->ScalarTolerance); + } + } + + // Fallback: direct OOC access + if(!isValidVoxel(point2)) + { + return false; + } + return m_CompareFunctor->compare(point1, point2); +} + +// ----------------------------------------------------------------------------- +// Allocates the rolling 2-slot slice buffers used by the CCL (OOC) algorithm. +// Called once at the start of the OOC branch in operator(), before executeCCL(). +// +// Each slot holds one full XY slice (dimX * dimY voxels). Two slots are needed +// because the CCL algorithm compares the current slice (iz) with the previous +// slice (iz-1), so both must be in memory simultaneously. +// +// Buffers allocated: +// - m_ScalarBuffer : 2 * sliceSize float64 values (one scalar per voxel, +// stored as float64 regardless of the original data type +// so that a single comparison path works for all types) +// - m_MaskBuffer : 2 * sliceSize uint8 values (one mask flag per voxel) +// +// Both m_BufferedSliceZ slots are initialized to -1 (no slice loaded). +// m_UseSliceBuffers is set to true so that isValidVoxel() and +// areNeighborsSimilar() will use the fast buffer path. +// ----------------------------------------------------------------------------- +void ScalarSegmentFeatures::allocateSliceBuffers(int64 dimX, int64 dimY) +{ + m_BufSliceSize = dimX * dimY; + const usize sliceSize = static_cast(m_BufSliceSize); + m_ScalarBuffer.resize(2 * sliceSize); + m_MaskBuffer.resize(2 * sliceSize); + m_BufferedSliceZ[0] = -1; + m_BufferedSliceZ[1] = -1; + m_UseSliceBuffers = true; +} + +// ----------------------------------------------------------------------------- +// Releases the slice buffers after executeCCL() completes, freeing the memory +// back to the system. Called in the OOC branch of operator() after the CCL +// algorithm finishes. Resets m_UseSliceBuffers to false and both +// m_BufferedSliceZ slots to -1. The vectors are replaced with default- +// constructed (empty) instances to guarantee memory deallocation. +// ----------------------------------------------------------------------------- +void ScalarSegmentFeatures::deallocateSliceBuffers() +{ + m_UseSliceBuffers = false; + m_ScalarBuffer = std::vector(); + m_MaskBuffer = std::vector(); + m_BufferedSliceZ[0] = -1; + m_BufferedSliceZ[1] = -1; +} + +// ----------------------------------------------------------------------------- +// Pre-loads voxel data for a single Z-slice into the rolling 2-slot buffer, +// called by executeCCL() before processing each slice. +// +// Rolling buffer design: +// The target slot is determined by (iz % 2), so even slices go to slot 0 and +// odd slices go to slot 1. Because the CCL algorithm processes slices in +// order (0, 1, 2, ...), at any given slice iz the previous slice (iz-1) is +// always in the other slot, keeping both the current and previous slice data +// available in memory. +// +// Sentinel behavior: +// If iz < 0, slice buffering is disabled (m_UseSliceBuffers = false). The +// CCL algorithm passes iz = -1 after completing the slice-by-slice sweep to +// signal that subsequent calls (e.g., during Phase 1b periodic boundary +// merging) should use direct DataStore access instead of the buffers. +// +// Skip-if-already-loaded: +// If m_BufferedSliceZ[slot] == iz, the data for this slice is already in the +// buffer (e.g., from a previous prepareForSlice call), so the method returns +// immediately without re-reading. +// +// Data loaded per slice: +// - Scalar values (1 float64 per voxel) into m_ScalarBuffer. The type +// dispatch uses ExecuteDataFunctionNoBool with FillScalarSliceBufferFunctor +// to convert the original typed data (int8..float64) to float64. Boolean +// arrays are handled separately because ExecuteDataFunctionNoBool excludes +// bool; they are converted to 0.0/1.0 manually. +// - Mask flags (1 uint8 per voxel) into m_MaskBuffer; if masking is disabled, +// all mask values are set to 1 (valid). +// ----------------------------------------------------------------------------- +void ScalarSegmentFeatures::prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) +{ + if(iz < 0) + { + m_UseSliceBuffers = false; + return; + } + if(!m_UseSliceBuffers) + { + return; + } + + const int slot = static_cast(iz % 2); + if(m_BufferedSliceZ[slot] == iz) + { + return; + } + + const usize sliceSize = static_cast(m_BufSliceSize); + const usize slotOffset = static_cast(slot) * sliceSize; + const int64 baseIndex = iz * m_BufSliceSize; + + // Fill scalar data buffer using type dispatch + DataType dataType = m_InputDataArray->getDataType(); + if(dataType == DataType::boolean) + { + auto& store = m_InputDataArray->template getIDataStoreRefAs>(); + // Bulk-read the entire boolean slice, then convert to float64 + auto boolBuf = std::make_unique(sliceSize); + store.copyIntoBuffer(static_cast(baseIndex), nonstd::span(boolBuf.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + m_ScalarBuffer[slotOffset + i] = boolBuf[i] ? 1.0 : 0.0; + } + } + else + { + ExecuteDataFunctionNoBool(FillScalarSliceBufferFunctor{}, dataType, m_InputDataArray, baseIndex, sliceSize, m_ScalarBuffer, slotOffset); + } + + // Fill mask buffer using bulk reads to avoid per-element OOC overhead + if(m_InputValues->UseMask && m_GoodVoxels != nullptr) + { + auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); + if(maskArray.getDataType() == DataType::uint8) + { + // Bulk-read uint8 mask data directly into the mask buffer + auto& typedStore = maskArray.getIDataStoreRefAs>(); + typedStore.copyIntoBuffer(static_cast(baseIndex), nonstd::span(m_MaskBuffer.data() + slotOffset, sliceSize)); + } + else if(maskArray.getDataType() == DataType::boolean) + { + // Bulk-read boolean mask data into a temp buffer, then convert to uint8 + auto& typedStore = maskArray.getIDataStoreRefAs>(); + auto boolBuf = std::make_unique(sliceSize); + typedStore.copyIntoBuffer(static_cast(baseIndex), nonstd::span(boolBuf.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + m_MaskBuffer[slotOffset + i] = boolBuf[i] ? 1 : 0; + } + } + else + { + // Fallback for unexpected mask types: per-element access via MaskCompare + for(usize i = 0; i < sliceSize; i++) + { + m_MaskBuffer[slotOffset + i] = m_GoodVoxels->isTrue(static_cast(baseIndex) + i) ? 1 : 0; + } + } + } + else + { + std::fill(m_MaskBuffer.begin() + slotOffset, m_MaskBuffer.begin() + slotOffset + sliceSize, static_cast(1)); + } + + m_BufferedSliceZ[slot] = iz; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp index 6a1393eeae..b8ae6fc94f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp @@ -51,32 +51,50 @@ class SIMPLNXCORE_EXPORT ScalarSegmentFeatures : public SegmentFeatures Result<> operator()(); protected: + int64 getSeed(int32 gnum, int64 nextSeed) const override; + bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; + /** - * @brief - * @param data - * @param args - * @param gnum - * @param nextSeed - * @return int64 + * @brief Checks whether a voxel can participate in scalar segmentation based on the mask. + * @param point Linear voxel index. + * @return true if the voxel passes the mask check (or no mask is used). */ - int64_t getSeed(int32 gnum, int64 nextSeed) const override; + bool isValidVoxel(int64 point) const override; /** - * @brief - * @param data - * @param args - * @param referencePoint - * @param neighborPoint - * @param gnum - * @return bool + * @brief Determines whether two neighboring voxels belong to the same scalar segment. + * @param point1 First voxel index. + * @param point2 Second (neighbor) voxel index. + * @return true if both voxels are valid and their scalar values are within tolerance. */ - bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; + bool areNeighborsSimilar(int64 point1, int64 point2) const override; + + /** + * @brief Pre-loads input scalar and mask data for the given Z-slice into + * rolling buffers, eliminating per-element OOC overhead during CCL. + * @param iz Current Z-slice index, or -1 to disable buffering. + * @param dimX X dimension of the grid. + * @param dimY Y dimension of the grid. + * @param dimZ Z dimension of the grid. + */ + void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) override; private: + void allocateSliceBuffers(int64 dimX, int64 dimY); + void deallocateSliceBuffers(); + const ScalarSegmentFeaturesInputValues* m_InputValues = nullptr; FeatureIdsArrayType* m_FeatureIdsArray = nullptr; GoodVoxelsArrayType* m_GoodVoxelsArray = nullptr; std::shared_ptr m_CompareFunctor; std::unique_ptr m_GoodVoxels = nullptr; + IDataArray* m_InputDataArray = nullptr; + + // Rolling 2-slot input buffers for OOC optimization. + std::vector m_ScalarBuffer; + std::vector m_MaskBuffer; + int64 m_BufSliceSize = 0; + int64 m_BufferedSliceZ[2] = {-1, -1}; + bool m_UseSliceBuffers = false; }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp new file mode 100644 index 0000000000..8476ed8943 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp @@ -0,0 +1,33 @@ +#include "SurfaceNets.hpp" +#include "SurfaceNetsDirect.hpp" +#include "SurfaceNetsScanline.hpp" + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" + +using namespace nx::core; + +// ----------------------------------------------------------------------------- +SurfaceNets::SurfaceNets(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +SurfaceNets::~SurfaceNets() noexcept = default; + +// ----------------------------------------------------------------------------- +const std::atomic_bool& SurfaceNets::getCancel() +{ + return m_ShouldCancel; +} + +// ----------------------------------------------------------------------------- +Result<> SurfaceNets::operator()() +{ + auto* featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); + return DispatchAlgorithm({featureIds}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp new file mode 100644 index 0000000000..27a7c621ef --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp @@ -0,0 +1,58 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "simplnx/DataStructure/DataPath.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/Parameters/MultiArraySelectionParameter.hpp" + +namespace nx::core +{ + +struct SIMPLNXCORE_EXPORT SurfaceNetsInputValues +{ + bool ApplySmoothing; + bool RepairTriangleWinding; + int32 SmoothingIterations; + float32 MaxDistanceFromVoxel; + float32 RelaxationFactor; + + DataPath GridGeomDataPath; + DataPath FeatureIdsArrayPath; + MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; + MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; + DataPath TriangleGeometryPath; + DataPath VertexGroupDataPath; + DataPath NodeTypesDataPath; + DataPath FaceGroupDataPath; + DataPath FaceLabelsDataPath; + MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; +}; + +/** + * @class + */ +class SIMPLNXCORE_EXPORT SurfaceNets +{ +public: + SurfaceNets(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues); + ~SurfaceNets() noexcept; + + SurfaceNets(const SurfaceNets&) = delete; + SurfaceNets(SurfaceNets&&) noexcept = delete; + SurfaceNets& operator=(const SurfaceNets&) = delete; + SurfaceNets& operator=(SurfaceNets&&) noexcept = delete; + + Result<> operator()(); + + const std::atomic_bool& getCancel(); + +private: + DataStructure& m_DataStructure; + const SurfaceNetsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp index 289d5ed44b..baae3c762a 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp @@ -1,4 +1,6 @@ #include "SurfaceNetsDirect.hpp" + +#include "SurfaceNets.hpp" #include "TupleTransfer.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -7,12 +9,15 @@ #include "simplnx/DataStructure/Geometry/TriangleGeom.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" #include "simplnx/Utilities/Meshing/TriangleUtilities.hpp" +#include "simplnx/Utilities/MessageHelper.hpp" #include "SimplnxCore/SurfaceNets/MMCellFlag.h" #include "SimplnxCore/SurfaceNets/MMCellMap.h" #include "SimplnxCore/SurfaceNets/MMGeometryOBJ.h" #include "SimplnxCore/SurfaceNets/MMSurfaceNet.h" +#include + using namespace nx::core; namespace @@ -23,7 +28,7 @@ constexpr inline int8 CalculatePadding(int8 value) return value + ((9 * static_cast(value < 10)) + 1); } -inline void HandlePadding(std::array vertexIndices, AbstractDataStore& nodeTypes) +inline void HandlePadding(std::array vertexIndices, AbstractDataStore& nodeTypes) { nodeTypes.setValue(vertexIndices[0], CalculatePadding(nodeTypes.getValue(vertexIndices[0]))); nodeTypes.setValue(vertexIndices[1], CalculatePadding(nodeTypes.getValue(vertexIndices[1]))); @@ -33,7 +38,7 @@ inline void HandlePadding(std::array vertexIndices, AbstractDataStore struct VertexData { - size_t VertexId; + usize VertexId = 0; std::array Position; }; @@ -44,18 +49,18 @@ void crossProduct(const std::array& vert0, const std::array& vert0, std::array& vert1, std::array& vert2) +float32 triangleArea(std::array& vert0, std::array& vert1, std::array& vert2) { // Area of triangle with vertex positions p0, p1, p2 const std::array v01 = {vert1[0] - vert0[0], vert1[1] - vert0[1], vert1[2] - vert0[2]}; const std::array v02 = {vert2[0] - vert0[0], vert2[1] - vert0[1], vert2[2] - vert0[2]}; std::array cross = {0.0f, 0.0f, 0.0f}; crossProduct(v01, v02, cross); - float const magCP = std::sqrt(cross[0] * cross[0] + cross[1] * cross[1] + cross[2] * cross[2]); + float32 const magCP = std::sqrt(cross[0] * cross[0] + cross[1] * cross[1] + cross[2] * cross[2]); return 0.5f * magCP; } -void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing, std::array& triangleVtxIDs) +void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing, std::array& triangleVtxIDs) { // Order quad vertices so quad is front facing if(!isQuadFrontFacing) @@ -67,8 +72,8 @@ void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing // Order quad vertices so that the two generated triangles have the minimal area. This // reduces self intersections in the surface. - float const thisArea = triangleArea(vData[0].Position, vData[1].Position, vData[2].Position) + triangleArea(vData[0].Position, vData[2].Position, vData[3].Position); - float const alternateArea = triangleArea(vData[1].Position, vData[2].Position, vData[3].Position) + triangleArea(vData[1].Position, vData[3].Position, vData[0].Position); + float32 const thisArea = triangleArea(vData[0].Position, vData[1].Position, vData[2].Position) + triangleArea(vData[0].Position, vData[2].Position, vData[3].Position); + float32 const alternateArea = triangleArea(vData[1].Position, vData[2].Position, vData[3].Position) + triangleArea(vData[1].Position, vData[3].Position, vData[0].Position); if(alternateArea < thisArea) { VertexData const temp = vData[0]; @@ -87,8 +92,9 @@ void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing triangleVtxIDs[5] = vData[3].VertexId; } } // namespace + // ----------------------------------------------------------------------------- -SurfaceNetsDirect::SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues) +SurfaceNetsDirect::SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const SurfaceNetsInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) , m_ShouldCancel(shouldCancel) @@ -99,15 +105,11 @@ SurfaceNetsDirect::SurfaceNetsDirect(DataStructure& dataStructure, const IFilter // ----------------------------------------------------------------------------- SurfaceNetsDirect::~SurfaceNetsDirect() noexcept = default; -// ----------------------------------------------------------------------------- -const std::atomic_bool& SurfaceNetsDirect::getCancel() -{ - return m_ShouldCancel; -} - // ----------------------------------------------------------------------------- Result<> SurfaceNetsDirect::operator()() { + MessageHelper messageHelper(m_MessageHandler); + // Get the ImageGeometry auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); @@ -119,15 +121,22 @@ Result<> SurfaceNetsDirect::operator()() auto voxelSize = imageGeom.getSpacing(); auto origin = imageGeom.getOrigin(); + messageHelper.sendMessage("Phase 1: Building surface net..."); MMSurfaceNet surfaceNet(triangleGeomPtr->getVerticesRef().getDataStoreRef(), m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath), gridDimensions.data(), voxelSize.data()); if(!surfaceNet.getCellMap()->valid()) { return MakeErrorResult(-843870, fmt::format("Could not allocate SurfaceNets internal data structures for grid geometry at path '{}'.", m_InputValues->GridGeomDataPath.toString())); } + if(m_ShouldCancel) + { + return {}; + } + // Use current parameters to relax the SurfaceNet if(m_InputValues->ApplySmoothing) { + messageHelper.sendMessage("Phase 2: Smoothing surface net..."); MMSurfaceNet::RelaxAttrs relaxAttrs{}; relaxAttrs.maxDistFromCellCenter = m_InputValues->MaxDistanceFromVoxel; relaxAttrs.numRelaxIterations = m_InputValues->SmoothingIterations; @@ -137,7 +146,7 @@ Result<> SurfaceNetsDirect::operator()() } auto cellMapPtr = surfaceNet.getCellMap(); - const size_t nodeCount = cellMapPtr->numVertices(); + const usize nodeCount = cellMapPtr->numVertices(); std::array arraySize2 = {0, 0, 0}; cellMapPtr->getArraySize(arraySize2.data()); @@ -148,11 +157,16 @@ Result<> SurfaceNetsDirect::operator()() auto& nodeTypes = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); nodeTypes.resizeTuples({static_cast(nodeCount)}); + messageHelper.sendMessage("Phase 3: Transforming vertex positions..."); Point3Df position = {0.0f, 0.0f, 0.0f}; std::array vertCellIndex = {0, 0, 0}; - for(size_t vertIndex = 0; vertIndex < nodeCount; vertIndex++) + for(usize vertIndex = 0; vertIndex < nodeCount; vertIndex++) { + if(m_ShouldCancel) + { + return {}; + } cellMapPtr->getVertexPosition(vertIndex, position.data()); // Relocate the vertex correctly based on the origin of the ImageGeometry position = position + origin - Point3Df(0.5f * voxelSize[0], 0.5f * voxelSize[1], 0.5f * voxelSize[1]); @@ -163,12 +177,17 @@ Result<> SurfaceNetsDirect::operator()() nodeTypes[static_cast(vertIndex)] = static_cast(currentCellPtr->flag.numJunctions()); } + messageHelper.sendMessage("Phase 4: Counting triangles..."); usize triangleCount = 0; std::array quadNxArrayIndices = {0, 0}; // First Pass through to just count the number of triangles: for(int idxVtx = 0; idxVtx < nodeCount; idxVtx++) { - std::array vertexIndices = {0, 0, 0, 0}; + if(m_ShouldCancel) + { + return {}; + } + std::array vertexIndices = {0, 0, 0, 0}; std::array<::LabelType, 2> quadLabels = {0, 0}; if(cellMapPtr->getEdgeQuad(idxVtx, MMCellFlag::Edge::BackBottomEdge, vertexIndices.data(), quadLabels.data(), quadNxArrayIndices.data())) @@ -206,7 +225,7 @@ Result<> SurfaceNetsDirect::operator()() // Create a vector of TupleTransferFunctions for each of the Triangle Face std::vector> tupleTransferFunctions; - for(size_t i = 0; i < m_InputValues->SelectedCellDataArrayPaths.size(); i++) + for(usize i = 0; i < m_InputValues->SelectedCellDataArrayPaths.size(); i++) { // Associate these arrays with the Triangle Face Data. ::AddTupleTransferInstance(m_DataStructure, m_InputValues->SelectedCellDataArrayPaths[i], m_InputValues->CreatedDataArrayPaths[i], tupleTransferFunctions); @@ -214,7 +233,7 @@ Result<> SurfaceNetsDirect::operator()() auto numSelectedCellArrayPaths = m_InputValues->SelectedCellDataArrayPaths.size(); - for(size_t i = 0; i < m_InputValues->SelectedFeatureDataArrayPaths.size(); i++) + for(usize i = 0; i < m_InputValues->SelectedFeatureDataArrayPaths.size(); i++) { // Associate these arrays with the Triangle Face Data. auto selectedPath = m_InputValues->SelectedFeatureDataArrayPaths[i]; @@ -222,20 +241,25 @@ Result<> SurfaceNetsDirect::operator()() ::AddFeatureTupleTransferInstance(m_DataStructure, selectedPath, createdPath, m_InputValues->FeatureIdsArrayPath, tupleTransferFunctions); } + messageHelper.sendMessage("Phase 5: Generating triangles..."); usize faceIndex = 0; // Create temporary storage for cell quads which are constructed around edges // crossed by the surface. Handle 3 edges per cell. The other 9 cell edges will // be handled when neighboring cells that share edges with this cell are visited. std::array t1 = {0, 0, 0}; std::array t2 = {0, 0, 0}; - std::array triangleVtxIDs = {0, 0, 0, 0, 0, 0}; - std::array vertexIndices = {0, 0, 0, 0}; + std::array triangleVtxIDs = {0, 0, 0, 0, 0, 0}; + std::array vertexIndices = {0, 0, 0, 0}; std::array quadLabels = {0, 0}; std::array vData{}; - std::array cellIndex = {0, 0, 0}; + std::array cellIndex = {0, 0, 0}; for(int idxVtx = 0; idxVtx < nodeCount; idxVtx++) { + if(m_ShouldCancel) + { + return {}; + } cellMapPtr->getVertexCellIndex(idxVtx, cellIndex.data()); // Back-bottom edge if(cellMapPtr->getEdgeQuad(idxVtx, MMCellFlag::Edge::BackBottomEdge, vertexIndices.data(), quadLabels.data(), quadNxArrayIndices.data())) @@ -427,6 +451,7 @@ Result<> SurfaceNetsDirect::operator()() Result<> windingResult = {}; if(m_InputValues->RepairTriangleWinding) { + messageHelper.sendMessage("Phase 6: Repairing triangle winding..."); // Generate Connectivity m_MessageHandler("Generating Connectivity and Triangle Neighbors..."); triangleGeom.findElementNeighbors(true); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp index 50aa0f8465..11a4d3db64 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp @@ -2,41 +2,23 @@ #include "SimplnxCore/SimplnxCore_export.hpp" -#include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/DataStructure.hpp" #include "simplnx/Filter/IFilter.hpp" -#include "simplnx/Parameters/MultiArraySelectionParameter.hpp" namespace nx::core { - -struct SIMPLNXCORE_EXPORT SurfaceNetsInputValues -{ - bool ApplySmoothing; - bool RepairTriangleWinding; - int32 SmoothingIterations; - float32 MaxDistanceFromVoxel; - float32 RelaxationFactor; - - DataPath GridGeomDataPath; - DataPath FeatureIdsArrayPath; - MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; - MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; - DataPath TriangleGeometryPath; - DataPath VertexGroupDataPath; - DataPath NodeTypesDataPath; - DataPath FaceGroupDataPath; - DataPath FaceLabelsDataPath; - MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; -}; +struct SurfaceNetsInputValues; /** - * @class + * @class SurfaceNetsDirect + * @brief In-core algorithm for SurfaceNets. Preserves the original sequential + * voxel iteration using MMSurfaceNet. Selected by DispatchAlgorithm when all + * input arrays are backed by in-memory DataStore. */ class SIMPLNXCORE_EXPORT SurfaceNetsDirect { public: - SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues); + SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const SurfaceNetsInputValues* inputValues); ~SurfaceNetsDirect() noexcept; SurfaceNetsDirect(const SurfaceNetsDirect&) = delete; @@ -46,8 +28,6 @@ class SIMPLNXCORE_EXPORT SurfaceNetsDirect Result<> operator()(); - const std::atomic_bool& getCancel(); - private: DataStructure& m_DataStructure; const SurfaceNetsInputValues* m_InputValues = nullptr; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp new file mode 100644 index 0000000000..73c4e6a81a --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp @@ -0,0 +1,893 @@ +#include "SurfaceNetsScanline.hpp" + +#include "SurfaceNets.hpp" +#include "TupleTransfer.hpp" + +#include "SimplnxCore/SurfaceNets/MMCellFlag.h" +#include "SimplnxCore/SurfaceNets/MMSurfaceNet.h" + +#include "simplnx/Common/Result.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataGroup.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/DataStructure/Geometry/TriangleGeom.hpp" +#include "simplnx/Utilities/DataArrayUtilities.hpp" +#include "simplnx/Utilities/Meshing/TriangleUtilities.hpp" + +#include + +#include +#include + +using namespace nx::core; + +namespace +{ +using LabelType = int32; + +/** + * @brief Packs a padded-grid (i,j,k) coordinate into a single uint64 key + * for the cell-to-vertex hash map. + */ +inline uint64 packCellKey(int32 i, int32 j, int32 k, int32 paddedX, int32 paddedXY) +{ + return static_cast(i) + static_cast(j) * static_cast(paddedX) + static_cast(k) * static_cast(paddedXY); +} + +constexpr inline int8 CalculatePadding(int8 value) +{ + return value + ((9 * static_cast(value < 10)) + 1); +} + +inline void HandlePadding(std::array vertexIndices, std::vector& nodeTypesBuf) +{ + nodeTypesBuf[vertexIndices[0]] = CalculatePadding(nodeTypesBuf[vertexIndices[0]]); + nodeTypesBuf[vertexIndices[1]] = CalculatePadding(nodeTypesBuf[vertexIndices[1]]); + nodeTypesBuf[vertexIndices[2]] = CalculatePadding(nodeTypesBuf[vertexIndices[2]]); + nodeTypesBuf[vertexIndices[3]] = CalculatePadding(nodeTypesBuf[vertexIndices[3]]); +} + +struct VertexData +{ + usize VertexId = 0; + std::array Position; +}; + +void crossProduct(const std::array& vert0, const std::array vert1, std::array result) +{ + // Cross product of vectors v0 and v1 + result[0] = vert0[1] * vert1[2] - vert0[2] * vert1[1]; + result[1] = vert0[2] * vert1[0] - vert0[0] * vert1[2]; + result[2] = vert0[0] * vert1[1] - vert0[1] * vert1[0]; +} + +float32 triangleArea(std::array& vert0, std::array& vert1, std::array& vert2) +{ + // Area of triangle with vertex positions p0, p1, p2 + const std::array v01 = {vert1[0] - vert0[0], vert1[1] - vert0[1], vert1[2] - vert0[2]}; + const std::array v02 = {vert2[0] - vert0[0], vert2[1] - vert0[1], vert2[2] - vert0[2]}; + std::array cross = {0.0f, 0.0f, 0.0f}; + crossProduct(v01, v02, cross); + float32 const magCP = std::sqrt(cross[0] * cross[0] + cross[1] * cross[1] + cross[2] * cross[2]); + return 0.5f * magCP; +} + +void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing, std::array& triangleVtxIDs) +{ + // Order quad vertices so quad is front facing + if(!isQuadFrontFacing) + { + VertexData const temp = vData[3]; + vData[3] = vData[1]; + vData[1] = temp; + } + + // Order quad vertices so that the two generated triangles have the minimal area. This + // reduces self intersections in the surface. + float32 const thisArea = triangleArea(vData[0].Position, vData[1].Position, vData[2].Position) + triangleArea(vData[0].Position, vData[2].Position, vData[3].Position); + float32 const alternateArea = triangleArea(vData[1].Position, vData[2].Position, vData[3].Position) + triangleArea(vData[1].Position, vData[3].Position, vData[0].Position); + if(alternateArea < thisArea) + { + VertexData const temp = vData[0]; + vData[0] = vData[1]; + vData[1] = vData[2]; + vData[2] = vData[3]; + vData[3] = temp; + } + + // Generate vertex ids to triangulate the quad + triangleVtxIDs[0] = vData[0].VertexId; + triangleVtxIDs[1] = vData[1].VertexId; + triangleVtxIDs[2] = vData[2].VertexId; + triangleVtxIDs[3] = vData[0].VertexId; + triangleVtxIDs[4] = vData[2].VertexId; + triangleVtxIDs[5] = vData[3].VertexId; +} + +/** + * @brief Looks up the vertex index for a neighboring cell at padded (ci,cj,ck) + * from the cell-to-vertex hash map. Returns max if not found (should not happen + * for valid edge quads). + */ +inline usize lookupVertex(const std::unordered_map& cellToVertex, int32 ci, int32 cj, int32 ck, int32 paddedX, int32 paddedXY) +{ + const uint64 key = packCellKey(ci, cj, ck, paddedX, paddedXY); + auto it = cellToVertex.find(key); + if(it != cellToVertex.end()) + { + return it->second; + } + return std::numeric_limits::max(); +} + +/** + * @brief Computes the label for a cell at padded coordinates (ci,cj,ck) + * by reading from the FeatureIds store. Boundary cells return Padding. + */ +inline int32 edgeCellLabel(int32 ci, int32 cj, int32 ck, int32 paddedX, int32 paddedY, int32 paddedZ, usize dimX, usize dimY, const AbstractDataStore& featureIdsStore) +{ + // Boundary padding check (same logic as MMCellMap::label) + if(ci <= 0 || cj <= 0 || ck <= 0 || ci >= paddedX - 1 || cj >= paddedY - 1 || ck >= paddedZ - 1) + { + return MMSurfaceNet::ReservedLabel::Padding; + } + // Convert padded to NX coordinates + const int32 nxX = ci - 1; + const int32 nxY = cj - 1; + const int32 nxZ = ck - 1; + // Additional range check (matches the checks in MMCellMap::label) + if(nxX < 0 || nxX >= static_cast(dimX) || nxY < 0 || nxY >= static_cast(dimY) || nxZ < 0) + { + return MMSurfaceNet::ReservedLabel::Padding; + } + const usize nxIdx = static_cast(nxZ) * dimY * dimX + static_cast(nxY) * dimX + static_cast(nxX); + return featureIdsStore.getValue(nxIdx); +} + +/** + * @brief Computes the flat NX array index for a cell at padded (ci,cj,ck). + * Returns max if outside the NX volume. + */ +inline usize edgeCellNxIndex(int32 ci, int32 cj, int32 ck, int32 paddedX, int32 paddedY, int32 paddedZ, usize dimX, usize dimY, usize dimZ) +{ + const int32 nxX = ci - 1; + const int32 nxY = cj - 1; + const int32 nxZ = ck - 1; + if(nxX < 0 || nxX >= static_cast(dimX) || nxY < 0 || nxY >= static_cast(dimY) || nxZ < 0 || nxZ >= static_cast(dimZ)) + { + return std::numeric_limits::max(); + } + return static_cast(nxZ) * dimY * dimX + static_cast(nxY) * dimX + static_cast(nxX); +} + +/** + * @brief Returns the FeatureId label for a corner at padded coordinates (ci,cj,ck). + * + * Boundary corners (any coordinate at 0 or >= paddedDim-1) return MMSurfaceNet::Padding. + * Interior corners look up the label from the appropriate slice buffer. + * + * @param ci Padded X coordinate of the corner + * @param cj Padded Y coordinate of the corner + * @param ck Padded Z coordinate of the corner + * @param paddedX Number of padded cells in X (dimX + 2) + * @param paddedY Number of padded cells in Y (dimY + 2) + * @param paddedZ Number of padded cells in Z (dimZ + 2) + * @param dimX Original NX dimension in X + * @param dimY Original NX dimension in Y + * @param slice0 Buffer holding NX Z-slice at index (currentK - 1) + * @param slice0Z The NX Z-index that slice0 currently holds + * @param slice1 Buffer holding NX Z-slice at index currentK + * @param slice1Z The NX Z-index that slice1 currently holds + */ +inline int32 cornerLabel(int32 ci, int32 cj, int32 ck, int32 paddedX, int32 paddedY, int32 paddedZ, usize dimX, usize dimY, const std::vector& slice0, int32 slice0Z, + const std::vector& slice1, int32 slice1Z) +{ + // Boundary padding check + if(ci <= 0 || cj <= 0 || ck <= 0 || ci >= paddedX - 1 || cj >= paddedY - 1 || ck >= paddedZ - 1) + { + return MMSurfaceNet::ReservedLabel::Padding; + } + + // Convert padded corner to NX coordinates + const int32 nxX = ci - 1; + const int32 nxY = cj - 1; + const int32 nxZ = ck - 1; + + // Compute the flat index within a single XY slice + const usize sliceOffset = static_cast(nxY) * dimX + static_cast(nxX); + + // Look up from the correct slice buffer + if(nxZ == slice0Z) + { + return slice0[sliceOffset]; + } + if(nxZ == slice1Z) + { + return slice1[sliceOffset]; + } + + // Should not reach here if slices are managed correctly + return MMSurfaceNet::ReservedLabel::Padding; +} +} // namespace + +// ----------------------------------------------------------------------------- +SurfaceNetsScanline::SurfaceNetsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const SurfaceNetsInputValues* inputValues) +: m_DataStructure(dataStructure) +, m_InputValues(inputValues) +, m_ShouldCancel(shouldCancel) +, m_MessageHandler(mesgHandler) +{ +} + +// ----------------------------------------------------------------------------- +SurfaceNetsScanline::~SurfaceNetsScanline() noexcept = default; + +// ----------------------------------------------------------------------------- +Result<> SurfaceNetsScanline::operator()() +{ + // ------------------------------------------------------------------------- + // 1. Get ImageGeom dimensions and compute padded dims + // ------------------------------------------------------------------------- + auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); + auto gridDimensions = imageGeom.getDimensions(); + + const usize dimX = gridDimensions[0]; + const usize dimY = gridDimensions[1]; + const usize dimZ = gridDimensions[2]; + + const int32 paddedX = static_cast(dimX) + 2; + const int32 paddedY = static_cast(dimY) + 2; + const int32 paddedZ = static_cast(dimZ) + 2; + const int32 paddedXY = paddedX * paddedY; + + // ------------------------------------------------------------------------- + // 2. Get the FeatureIds DataStore reference + // ------------------------------------------------------------------------- + auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + + // ------------------------------------------------------------------------- + // 3. Allocate rolling slice buffers (2 NX Z-slices) + // ------------------------------------------------------------------------- + const usize sliceSize = dimX * dimY; + std::vector sliceBufA(sliceSize); + std::vector sliceBufB(sliceSize); + + // Pointers for ping-pong: slice0 is the "lower" Z-slice, slice1 is "upper" + std::vector* slice0 = &sliceBufA; + std::vector* slice1 = &sliceBufB; + int32 slice0Z = -1; + int32 slice1Z = -1; + + // ------------------------------------------------------------------------- + // 4. Iterate cells in the same order as setCellVertices(): + // k in [0, paddedZ-2), j in [0, paddedY-2), i in [0, paddedX-2) + // ------------------------------------------------------------------------- + const usize totalPaddedZSlices = static_cast(paddedZ - 1); + for(int32 k = 0; k < paddedZ - 1; k++) + { + if(m_ShouldCancel) + { + return {}; + } + // The 8 corners of a cell at padded (i,j,k) span NX Z-indices: + // bottom face corners: ck = k => nxZ = k - 1 + // top face corners: ck = k + 1 => nxZ = k + // So we need NX slices at Z = (k-1) and Z = k. + const int32 needZ0 = k - 1; // NX Z for bottom face corners + const int32 needZ1 = k; // NX Z for top face corners + + // Load the two required slices + if(needZ0 >= 0 && needZ0 < static_cast(dimZ)) + { + if(slice0Z != needZ0 && slice1Z != needZ0) + { + // Load needZ0 into slice0 + featureIdsStore.copyIntoBuffer(static_cast(needZ0) * sliceSize, nonstd::span(slice0->data(), sliceSize)); + slice0Z = needZ0; + } + } + if(needZ1 >= 0 && needZ1 < static_cast(dimZ)) + { + if(slice0Z != needZ1 && slice1Z != needZ1) + { + // Load needZ1 into slice1 + featureIdsStore.copyIntoBuffer(static_cast(needZ1) * sliceSize, nonstd::span(slice1->data(), sliceSize)); + slice1Z = needZ1; + } + } + + for(int32 j = 0; j < paddedY - 1; j++) + { + for(int32 i = 0; i < paddedX - 1; i++) + { + // Compute 8 corner labels for cell at padded (i,j,k) + // Corner ordering matches MMCellMap::setCellVertices(): + // [0] = (i, j, k ) left-back-bottom + // [1] = (i+1, j, k ) right-back-bottom + // [2] = (i+1, j+1, k ) right-front-bottom + // [3] = (i, j+1, k ) left-front-bottom + // [4] = (i, j, k+1) left-back-top + // [5] = (i+1, j, k+1) right-back-top + // [6] = (i+1, j+1, k+1) right-front-top + // [7] = (i, j+1, k+1) left-front-top + int32 cellLabels[8]; + + cellLabels[0] = cornerLabel(i, j, k, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[1] = cornerLabel(i + 1, j, k, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[2] = cornerLabel(i + 1, j + 1, k, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[3] = cornerLabel(i, j + 1, k, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[4] = cornerLabel(i, j, k + 1, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[5] = cornerLabel(i + 1, j, k + 1, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[6] = cornerLabel(i + 1, j + 1, k + 1, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + cellLabels[7] = cornerLabel(i, j + 1, k + 1, paddedX, paddedY, paddedZ, dimX, dimY, *slice0, slice0Z, *slice1, slice1Z); + + MMCellFlag flag; + flag.clear(); + flag.set(cellLabels); + + if(flag.vertexType() != MMCellFlag::VertexType::NoVertex) + { + const usize vertexIndex = m_Vertices.size(); + m_Vertices.push_back(VertexInfo{{i, j, k}, flag}); + m_CellToVertex[packCellKey(i, j, k, paddedX, paddedXY)] = vertexIndex; + } + } + } + + // Roll slice buffers for the next k iteration. + // After processing k, the "bottom" slice (needZ0 = k-1) is no longer + // needed for k+1 (which will need k and k+1). So we swap so that + // the current "top" slice becomes the new "bottom" and the old + // "bottom" buffer is available for loading. + std::swap(slice0, slice1); + std::swap(slice0Z, slice1Z); + } + + // ------------------------------------------------------------------------- + // 5. Resize TriangleGeom vertex array and associated attribute arrays + // ------------------------------------------------------------------------- + const usize numVertices = m_Vertices.size(); + + auto& triangleGeom = m_DataStructure.getDataRefAs(m_InputValues->TriangleGeometryPath); + auto& verticesStore = triangleGeom.getVerticesRef().getDataStoreRef(); + verticesStore.resizeTuples(ShapeType{numVertices}); + + triangleGeom.getVertexAttributeMatrix()->resizeTuples({numVertices}); + + auto& nodeTypes = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); + nodeTypes.resizeTuples({numVertices}); + + // NodeTypes buffer -- filled during Phase 2B, modified during Phase 3A, + // then flushed once via copyFromBuffer before triangle generation. + std::vector nodeTypesBuf(numVertices, 0); + + // ------------------------------------------------------------------------- + // 6. Phase 2A: Relaxation (optional — only if smoothing is requested) + // ------------------------------------------------------------------------- + // Face direction offsets: Left(-1,0,0), Right(+1,0,0), Back(0,-1,0), + // Front(0,+1,0), Bottom(0,0,-1), Top(0,0,+1) + static constexpr int32 k_FaceOffsets[6][3] = { + {-1, 0, 0}, // LeftFace + {+1, 0, 0}, // RightFace + {0, -1, 0}, // BackFace + {0, +1, 0}, // FrontFace + {0, 0, -1}, // BottomFace + {0, 0, +1} // TopFace + }; + + // Use a local buffer for all position work. Initial value is (0.5, 0.5, 0.5) + // -- cell center in local coords. This avoids O(iterations * vertices * 6) + std::vector localPos(3 * numVertices); + for(usize v = 0; v < numVertices; v++) + { + localPos[v * 3 + 0] = 0.5f; + localPos[v * 3 + 1] = 0.5f; + localPos[v * 3 + 2] = 0.5f; + } + + if(m_InputValues->ApplySmoothing) + { + const float32 alpha = m_InputValues->RelaxationFactor; + const float32 maxDist = m_InputValues->MaxDistanceFromVoxel; + const float32 minClamp = 0.5f - maxDist; + const float32 maxClamp = 0.5f + maxDist; + + const usize totalSmoothingIterations = static_cast(m_InputValues->SmoothingIterations); + for(int32 iter = 0; iter < m_InputValues->SmoothingIterations; iter++) + { + if(m_ShouldCancel) + { + return {}; + } + for(usize v = 0; v < numVertices; v++) + { + const auto& vi = m_Vertices[v]; + const int32 cellI = vi.cellIndex[0]; + const int32 cellJ = vi.cellIndex[1]; + const int32 cellK = vi.cellIndex[2]; + + int32 numNeighbors = 0; + float32 avgP[3] = {0.0f, 0.0f, 0.0f}; + + for(MMCellFlag::Face face = MMCellFlag::Face::LeftFace; face <= MMCellFlag::Face::TopFace; ++face) + { + // Determine whether this face participates based on vertex type + bool participates = false; + if(vi.flag.vertexType() == MMCellFlag::VertexType::SurfaceVertex) + { + participates = (vi.flag.faceCrossingType(face) != MMCellFlag::FaceCrossingType::NoFaceCrossing); + } + else + { + participates = (vi.flag.faceCrossingType(face) == MMCellFlag::FaceCrossingType::JunctionFaceCrossing); + } + + if(!participates) + { + continue; + } + + const int32 faceIdx = static_cast(face); + const int32 nbrI = cellI + k_FaceOffsets[faceIdx][0]; + const int32 nbrJ = cellJ + k_FaceOffsets[faceIdx][1]; + const int32 nbrK = cellK + k_FaceOffsets[faceIdx][2]; + + // Look up neighbor vertex in the hash map + const uint64 nbrKey = packCellKey(nbrI, nbrJ, nbrK, paddedX, paddedXY); + auto it = m_CellToVertex.find(nbrKey); + + float32 nbrPosX = 0.0f, nbrPosY = 0.0f, nbrPosZ = 0.0f; + if(it != m_CellToVertex.end()) + { + const usize nbrVtxIdx = it->second; + nbrPosX = localPos[nbrVtxIdx * 3 + 0]; + nbrPosY = localPos[nbrVtxIdx * 3 + 1]; + nbrPosZ = localPos[nbrVtxIdx * 3 + 2]; + } + else + { + // Neighbor cell has no vertex — use default position (0.5) + nbrPosX = 0.5f; + nbrPosY = 0.5f; + nbrPosZ = 0.5f; + } + + // Accumulate: neighbor position + offset from current cell to neighbor cell + avgP[0] += nbrPosX + static_cast(nbrI - cellI); + avgP[1] += nbrPosY + static_cast(nbrJ - cellJ); + avgP[2] += nbrPosZ + static_cast(nbrK - cellK); + numNeighbors++; + } + + // Blend current position with average neighbor position + if(numNeighbors > 0) + { + avgP[0] /= static_cast(numNeighbors); + avgP[1] /= static_cast(numNeighbors); + avgP[2] /= static_cast(numNeighbors); + + float32 x = (1.0f - alpha) * localPos[v * 3 + 0] + alpha * avgP[0]; + x = std::clamp(x, minClamp, maxClamp); + localPos[v * 3 + 0] = x; + + float32 y = (1.0f - alpha) * localPos[v * 3 + 1] + alpha * avgP[1]; + y = std::clamp(y, minClamp, maxClamp); + localPos[v * 3 + 1] = y; + + float32 z = (1.0f - alpha) * localPos[v * 3 + 2] + alpha * avgP[2]; + z = std::clamp(z, minClamp, maxClamp); + localPos[v * 3 + 2] = z; + } + } + } + } + + // ------------------------------------------------------------------------- + // 7. Phase 2B: Transform vertex positions to world coordinates and assign + // NodeTypes. Replicates SurfaceNetsDirect.cpp lines 148-161. + // ------------------------------------------------------------------------- + auto voxelSize = imageGeom.getSpacing(); + auto origin = imageGeom.getOrigin(); + // Note: the Z offset intentionally uses voxelSize[1] to match the original + // SurfaceNetsDirect.cpp behavior (line 155). + const Point3Df halfVoxelOffset(0.5f * voxelSize[0], 0.5f * voxelSize[1], 0.5f * voxelSize[1]); + + // Build vertex positions in localPos (reusing the smoothing buffer) and + // nodeTypes in nodeTypesBuf, then flush both with bulk copyFromBuffer. + for(usize v = 0; v < numVertices; v++) + { + if(m_ShouldCancel) + { + return {}; + } + const auto& vi = m_Vertices[v]; + + localPos[v * 3 + 0] = voxelSize[0] * (static_cast(vi.cellIndex[0]) + localPos[v * 3 + 0]) + origin[0] - halfVoxelOffset[0]; + localPos[v * 3 + 1] = voxelSize[1] * (static_cast(vi.cellIndex[1]) + localPos[v * 3 + 1]) + origin[1] - halfVoxelOffset[1]; + localPos[v * 3 + 2] = voxelSize[2] * (static_cast(vi.cellIndex[2]) + localPos[v * 3 + 2]) + origin[2] - halfVoxelOffset[2]; + + nodeTypesBuf[v] = static_cast(vi.flag.numJunctions()); + } + + // ------------------------------------------------------------------------- + // 8. Phase 3A: First pass — count triangles + // ------------------------------------------------------------------------- + usize triangleCount = 0; + std::array quadNxArrayIndices = {0, 0}; + + for(usize v = 0; v < numVertices; v++) + { + if(m_ShouldCancel) + { + return {}; + } + + const auto& vi = m_Vertices[v]; + const int32 cellI = vi.cellIndex[0]; + const int32 cellJ = vi.cellIndex[1]; + const int32 cellK = vi.cellIndex[2]; + + // BackBottomEdge + if(vi.flag.isEdgeCrossing(MMCellFlag::Edge::BackBottomEdge)) + { + std::array vertexIndices = {v, lookupVertex(m_CellToVertex, cellI, cellJ - 1, cellK, paddedX, paddedXY), lookupVertex(m_CellToVertex, cellI, cellJ - 1, cellK - 1, paddedX, paddedXY), + lookupVertex(m_CellToVertex, cellI, cellJ, cellK - 1, paddedX, paddedXY)}; + + std::array quadLabels; + quadLabels[0] = edgeCellLabel(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + quadLabels[1] = edgeCellLabel(cellI + 1, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + + if(quadLabels[0] == MMSurfaceNet::Padding || quadLabels[1] == MMSurfaceNet::Padding) + { + HandlePadding(vertexIndices, nodeTypesBuf); + } + triangleCount += 2; + } + + // LeftBottomEdge + if(vi.flag.isEdgeCrossing(MMCellFlag::Edge::LeftBottomEdge)) + { + std::array vertexIndices = {v, lookupVertex(m_CellToVertex, cellI, cellJ, cellK - 1, paddedX, paddedXY), lookupVertex(m_CellToVertex, cellI - 1, cellJ, cellK - 1, paddedX, paddedXY), + lookupVertex(m_CellToVertex, cellI - 1, cellJ, cellK, paddedX, paddedXY)}; + + std::array quadLabels; + quadLabels[0] = edgeCellLabel(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + quadLabels[1] = edgeCellLabel(cellI, cellJ + 1, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + + if(quadLabels[0] == MMSurfaceNet::Padding || quadLabels[1] == MMSurfaceNet::Padding) + { + HandlePadding(vertexIndices, nodeTypesBuf); + } + triangleCount += 2; + } + + // LeftBackEdge + if(vi.flag.isEdgeCrossing(MMCellFlag::Edge::LeftBackEdge)) + { + std::array vertexIndices = {v, lookupVertex(m_CellToVertex, cellI - 1, cellJ, cellK, paddedX, paddedXY), lookupVertex(m_CellToVertex, cellI - 1, cellJ - 1, cellK, paddedX, paddedXY), + lookupVertex(m_CellToVertex, cellI, cellJ - 1, cellK, paddedX, paddedXY)}; + + std::array quadLabels; + quadLabels[0] = edgeCellLabel(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + quadLabels[1] = edgeCellLabel(cellI, cellJ, cellK + 1, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + + if(quadLabels[0] == MMSurfaceNet::Padding || quadLabels[1] == MMSurfaceNet::Padding) + { + HandlePadding(vertexIndices, nodeTypesBuf); + } + triangleCount += 2; + } + } + + // Flush buffered vertex positions and nodeTypes to their DataStores + verticesStore.copyFromBuffer(0, nonstd::span(localPos.data(), localPos.size())); + nodeTypes.copyFromBuffer(0, nonstd::span(nodeTypesBuf.data(), nodeTypesBuf.size())); + + // ------------------------------------------------------------------------- + // 9. Phase 3B: Resize face arrays + // ------------------------------------------------------------------------- + triangleGeom.resizeFaceList(triangleCount); + triangleGeom.getFaceAttributeMatrix()->resizeTuples({triangleCount}); + + auto& faceLabels = m_DataStructure.getDataAs(m_InputValues->FaceLabelsDataPath)->getDataStoreRef(); + faceLabels.resizeTuples({triangleCount}); + + // ------------------------------------------------------------------------- + // 10. Phase 3C: TupleTransfer setup + // ------------------------------------------------------------------------- + std::vector> tupleTransferFunctions; + for(usize i = 0; i < m_InputValues->SelectedCellDataArrayPaths.size(); i++) + { + ::AddTupleTransferInstance(m_DataStructure, m_InputValues->SelectedCellDataArrayPaths[i], m_InputValues->CreatedDataArrayPaths[i], tupleTransferFunctions); + } + + auto numSelectedCellArrayPaths = m_InputValues->SelectedCellDataArrayPaths.size(); + + for(usize i = 0; i < m_InputValues->SelectedFeatureDataArrayPaths.size(); i++) + { + auto selectedPath = m_InputValues->SelectedFeatureDataArrayPaths[i]; + auto createdPath = m_InputValues->CreatedDataArrayPaths[i + numSelectedCellArrayPaths]; + ::AddFeatureTupleTransferInstance(m_DataStructure, selectedPath, createdPath, m_InputValues->FeatureIdsArrayPath, tupleTransferFunctions); + } + + // ------------------------------------------------------------------------- + // 11. Phase 3D: Second pass — generate triangles + // ------------------------------------------------------------------------- + using MeshIndexType = IGeometry::MeshIndexType; + auto& facesStore = triangleGeom.getFacesRef().getDataStoreRef(); + + std::vector triConnBuf; + std::vector faceLabelBuf; + std::vector ttArgsBuf; + triConnBuf.reserve(triangleCount * 3); + faceLabelBuf.reserve(triangleCount * 2); + ttArgsBuf.reserve(triangleCount); + + usize faceIndex = 0; + std::array triangleVtxIDs = {0, 0, 0, 0, 0, 0}; + std::array vertexIndices = {0, 0, 0, 0}; + std::array quadLabels = {0, 0}; + std::array vData{}; + + for(usize v = 0; v < numVertices; v++) + { + if(m_ShouldCancel) + { + return {}; + } + const auto& vi = m_Vertices[v]; + const int32 cellI = vi.cellIndex[0]; + const int32 cellJ = vi.cellIndex[1]; + const int32 cellK = vi.cellIndex[2]; + + // Back-bottom edge + if(vi.flag.isEdgeCrossing(MMCellFlag::Edge::BackBottomEdge)) + { + vertexIndices[0] = v; + vertexIndices[1] = lookupVertex(m_CellToVertex, cellI, cellJ - 1, cellK, paddedX, paddedXY); + vertexIndices[2] = lookupVertex(m_CellToVertex, cellI, cellJ - 1, cellK - 1, paddedX, paddedXY); + vertexIndices[3] = lookupVertex(m_CellToVertex, cellI, cellJ, cellK - 1, paddedX, paddedXY); + + quadLabels[0] = edgeCellLabel(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + quadLabels[1] = edgeCellLabel(cellI + 1, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + + quadNxArrayIndices[0] = edgeCellNxIndex(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, dimZ); + quadNxArrayIndices[1] = edgeCellNxIndex(cellI + 1, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, dimZ); + + vData[0] = {vertexIndices[0], {0.0f, 0.0f, 0.0f}}; + vData[1] = {vertexIndices[1], {0.0f, 0.0f, 0.0f}}; + vData[2] = {vertexIndices[2], {0.0f, 0.0f, 0.0f}}; + vData[3] = {vertexIndices[3], {0.0f, 0.0f, 0.0f}}; + + const bool isQuadFrontFacing = (quadLabels[0] < quadLabels[1]); + if(quadLabels[0] == MMSurfaceNet::Padding) + { + quadLabels[0] = 0; + } + if(quadLabels[1] == MMSurfaceNet::Padding) + { + quadLabels[1] = 0; + } + + getQuadTriangleIDs(vData, isQuadFrontFacing, triangleVtxIDs); + + // Triangle 1 + triConnBuf.push_back(triangleVtxIDs[0]); + triConnBuf.push_back(triangleVtxIDs[1]); + triConnBuf.push_back(triangleVtxIDs[2]); + if(quadLabels[0] < quadLabels[1]) + { + faceLabelBuf.push_back(quadLabels[0]); + faceLabelBuf.push_back(quadLabels[1]); + } + else + { + faceLabelBuf.push_back(quadLabels[1]); + faceLabelBuf.push_back(quadLabels[0]); + } + ttArgsBuf.push_back({faceIndex, quadNxArrayIndices}); + faceIndex++; + + // Triangle 2 + triConnBuf.push_back(triangleVtxIDs[3]); + triConnBuf.push_back(triangleVtxIDs[4]); + triConnBuf.push_back(triangleVtxIDs[5]); + if(quadLabels[0] < quadLabels[1]) + { + faceLabelBuf.push_back(quadLabels[0]); + faceLabelBuf.push_back(quadLabels[1]); + } + else + { + faceLabelBuf.push_back(quadLabels[1]); + faceLabelBuf.push_back(quadLabels[0]); + } + ttArgsBuf.push_back({faceIndex, quadNxArrayIndices}); + faceIndex++; + } + + // Left-bottom edge + if(vi.flag.isEdgeCrossing(MMCellFlag::Edge::LeftBottomEdge)) + { + vertexIndices[0] = v; + vertexIndices[1] = lookupVertex(m_CellToVertex, cellI, cellJ, cellK - 1, paddedX, paddedXY); + vertexIndices[2] = lookupVertex(m_CellToVertex, cellI - 1, cellJ, cellK - 1, paddedX, paddedXY); + vertexIndices[3] = lookupVertex(m_CellToVertex, cellI - 1, cellJ, cellK, paddedX, paddedXY); + + quadLabels[0] = edgeCellLabel(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + quadLabels[1] = edgeCellLabel(cellI, cellJ + 1, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + + quadNxArrayIndices[0] = edgeCellNxIndex(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, dimZ); + quadNxArrayIndices[1] = edgeCellNxIndex(cellI, cellJ + 1, cellK, paddedX, paddedY, paddedZ, dimX, dimY, dimZ); + + vData[0] = {vertexIndices[0], {0.0f, 0.0f, 0.0f}}; + vData[1] = {vertexIndices[1], {0.0f, 0.0f, 0.0f}}; + vData[2] = {vertexIndices[2], {0.0f, 0.0f, 0.0f}}; + vData[3] = {vertexIndices[3], {0.0f, 0.0f, 0.0f}}; + + const bool isQuadFrontFacing = (quadLabels[0] < quadLabels[1]); + if(quadLabels[0] == MMSurfaceNet::Padding) + { + quadLabels[0] = 0; + } + if(quadLabels[1] == MMSurfaceNet::Padding) + { + quadLabels[1] = 0; + } + + getQuadTriangleIDs(vData, isQuadFrontFacing, triangleVtxIDs); + + // Triangle 1 + triConnBuf.push_back(triangleVtxIDs[0]); + triConnBuf.push_back(triangleVtxIDs[1]); + triConnBuf.push_back(triangleVtxIDs[2]); + if(quadLabels[0] < quadLabels[1]) + { + faceLabelBuf.push_back(quadLabels[0]); + faceLabelBuf.push_back(quadLabels[1]); + } + else + { + faceLabelBuf.push_back(quadLabels[1]); + faceLabelBuf.push_back(quadLabels[0]); + } + ttArgsBuf.push_back({faceIndex, quadNxArrayIndices}); + faceIndex++; + + // Triangle 2 + triConnBuf.push_back(triangleVtxIDs[3]); + triConnBuf.push_back(triangleVtxIDs[4]); + triConnBuf.push_back(triangleVtxIDs[5]); + if(quadLabels[0] < quadLabels[1]) + { + faceLabelBuf.push_back(quadLabels[0]); + faceLabelBuf.push_back(quadLabels[1]); + } + else + { + faceLabelBuf.push_back(quadLabels[1]); + faceLabelBuf.push_back(quadLabels[0]); + } + ttArgsBuf.push_back({faceIndex, quadNxArrayIndices}); + faceIndex++; + } + + // Left-back edge + if(vi.flag.isEdgeCrossing(MMCellFlag::Edge::LeftBackEdge)) + { + vertexIndices[0] = v; + vertexIndices[1] = lookupVertex(m_CellToVertex, cellI - 1, cellJ, cellK, paddedX, paddedXY); + vertexIndices[2] = lookupVertex(m_CellToVertex, cellI - 1, cellJ - 1, cellK, paddedX, paddedXY); + vertexIndices[3] = lookupVertex(m_CellToVertex, cellI, cellJ - 1, cellK, paddedX, paddedXY); + + quadLabels[0] = edgeCellLabel(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + quadLabels[1] = edgeCellLabel(cellI, cellJ, cellK + 1, paddedX, paddedY, paddedZ, dimX, dimY, featureIdsStore); + + quadNxArrayIndices[0] = edgeCellNxIndex(cellI, cellJ, cellK, paddedX, paddedY, paddedZ, dimX, dimY, dimZ); + quadNxArrayIndices[1] = edgeCellNxIndex(cellI, cellJ, cellK + 1, paddedX, paddedY, paddedZ, dimX, dimY, dimZ); + + vData[0] = {vertexIndices[0], {0.0f, 0.0f, 0.0f}}; + vData[1] = {vertexIndices[1], {0.0f, 0.0f, 0.0f}}; + vData[2] = {vertexIndices[2], {0.0f, 0.0f, 0.0f}}; + vData[3] = {vertexIndices[3], {0.0f, 0.0f, 0.0f}}; + + const bool isQuadFrontFacing = (quadLabels[0] < quadLabels[1]); + if(quadLabels[0] == MMSurfaceNet::Padding) + { + quadLabels[0] = 0; + } + if(quadLabels[1] == MMSurfaceNet::Padding) + { + quadLabels[1] = 0; + } + + getQuadTriangleIDs(vData, isQuadFrontFacing, triangleVtxIDs); + + // Triangle 1 + triConnBuf.push_back(triangleVtxIDs[0]); + triConnBuf.push_back(triangleVtxIDs[1]); + triConnBuf.push_back(triangleVtxIDs[2]); + if(quadLabels[0] < quadLabels[1]) + { + faceLabelBuf.push_back(quadLabels[0]); + faceLabelBuf.push_back(quadLabels[1]); + } + else + { + faceLabelBuf.push_back(quadLabels[1]); + faceLabelBuf.push_back(quadLabels[0]); + } + ttArgsBuf.push_back({faceIndex, quadNxArrayIndices}); + faceIndex++; + + // Triangle 2 + triConnBuf.push_back(triangleVtxIDs[3]); + triConnBuf.push_back(triangleVtxIDs[4]); + triConnBuf.push_back(triangleVtxIDs[5]); + if(quadLabels[0] < quadLabels[1]) + { + faceLabelBuf.push_back(quadLabels[0]); + faceLabelBuf.push_back(quadLabels[1]); + } + else + { + faceLabelBuf.push_back(quadLabels[1]); + faceLabelBuf.push_back(quadLabels[0]); + } + ttArgsBuf.push_back({faceIndex, quadNxArrayIndices}); + faceIndex++; + } + } + + // ------------------------------------------------------------------------- + // 12. Phase 3E: FaceLabels fixup -- replace 0 with -1 in the local buffer + // ------------------------------------------------------------------------- + for(auto& label : faceLabelBuf) + { + if(label == 0) + { + label = -1; + } + } + + // ------------------------------------------------------------------------- + // 12b. Flush buffered triangle connectivity, face labels, and TupleTransfer + // ------------------------------------------------------------------------- + if(!triConnBuf.empty()) + { + facesStore.copyFromBuffer(0, nonstd::span(triConnBuf.data(), triConnBuf.size())); + faceLabels.copyFromBuffer(0, nonstd::span(faceLabelBuf.data(), faceLabelBuf.size())); + for(const auto& tupleTransferFunction : tupleTransferFunctions) + { + tupleTransferFunction->surfaceNetsTransferBatch(nonstd::span(ttArgsBuf.data(), ttArgsBuf.size())); + } + } + + // ------------------------------------------------------------------------- + // 13. Phase 3F: Winding repair + // ------------------------------------------------------------------------- + Result<> windingResult = {}; + if(m_InputValues->RepairTriangleWinding) + { + m_MessageHandler("Generating Connectivity and Triangle Neighbors..."); + triangleGeom.findElementNeighbors(true); + const auto optionalId = triangleGeom.getElementNeighborsId(); + if(!optionalId.has_value()) + { + return MakeErrorResult(-56331, fmt::format("Unable to generate the connectivity list for {} geometry.", triangleGeom.getName())); + } + const auto& connectivity = m_DataStructure.getDataRefAs(optionalId.value()); + + m_MessageHandler("Repairing Windings..."); + + windingResult = MeshingUtilities::RepairTriangleWinding(triangleGeom.getFaces()->getDataStoreRef(), connectivity, + m_DataStructure.getDataAs(m_InputValues->FaceLabelsDataPath)->getDataStoreRef(), m_ShouldCancel, m_MessageHandler); + + // Purge connectivity + m_DataStructure.removeData(triangleGeom.getElementContainingVertId().value()); + m_DataStructure.removeData(triangleGeom.getElementNeighborsId().value()); + } + + return windingResult; +} diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp new file mode 100644 index 0000000000..488d65d0a1 --- /dev/null +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp @@ -0,0 +1,60 @@ +#pragma once + +#include "SimplnxCore/SimplnxCore_export.hpp" + +#include "SimplnxCore/SurfaceNets/MMCellFlag.h" + +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/Filter/IFilter.hpp" + +#include +#include +#include + +namespace nx::core +{ +struct SurfaceNetsInputValues; + +/** + * @class SurfaceNetsScanline + * @brief Out-of-core algorithm for SurfaceNets. Selected by DispatchAlgorithm + * when any input array is backed by chunked (OOC) storage. + * + * Phase 1 performs slice-by-slice cell classification, reading FeatureIds + * via copyIntoBuffer in Z-slices. Surface cells are stored in O(surface) + * data structures rather than the O(n) Cell array used by MMCellMap. + */ +class SIMPLNXCORE_EXPORT SurfaceNetsScanline +{ +public: + SurfaceNetsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const SurfaceNetsInputValues* inputValues); + ~SurfaceNetsScanline() noexcept; + + SurfaceNetsScanline(const SurfaceNetsScanline&) = delete; + SurfaceNetsScanline(SurfaceNetsScanline&&) noexcept = delete; + SurfaceNetsScanline& operator=(const SurfaceNetsScanline&) = delete; + SurfaceNetsScanline& operator=(SurfaceNetsScanline&&) noexcept = delete; + + Result<> operator()(); + + /** + * @brief Per-vertex information stored only for surface cells. + */ + struct VertexInfo + { + std::array cellIndex; // (i,j,k) in padded coordinates + MMCellFlag flag; + }; + +private: + DataStructure& m_DataStructure; + const SurfaceNetsInputValues* m_InputValues = nullptr; + const std::atomic_bool& m_ShouldCancel; + const IFilter::MessageHandler& m_MessageHandler; + + // O(surface) data structures — populated during cell classification + std::vector m_Vertices; + std::unordered_map m_CellToVertex; +}; + +} // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp index adb92820cb..521148a2f3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp @@ -7,9 +7,25 @@ #include "simplnx/DataStructure/DataStructure.hpp" #include +#include namespace nx::core { +struct QuickSurfaceTransferData +{ + usize faceIndex = 0; + usize firstcIndex = 0; + usize secondcIndex = 0; + int32 faceLabel0 = 0; + int32 faceLabel1 = 0; +}; + +struct SurfaceNetsTransferData +{ + usize faceIndex = 0; + std::array quadNxArrayIndices = {std::numeric_limits::max(), std::numeric_limits::max()}; +}; + /** * @brief This is the base class that is used to transfer cell data to triangle face data * but could be used generally to copy the tuple value from one Data Array to another @@ -48,6 +64,14 @@ class SIMPLNXCORE_EXPORT AbstractTupleTransfer */ virtual void surfaceNetsTransfer(size_t faceIndex, const std::array& quadNxArrayIndices) = 0; + virtual void quickSurfaceTransferBatch(nonstd::span /*records*/) + { + } + + virtual void surfaceNetsTransferBatch(nonstd::span /*records*/) + { + } + protected: AbstractTupleTransfer() = default; @@ -153,6 +177,146 @@ class TransferTuple : public AbstractTupleTransfer } } + void quickSurfaceTransferBatch(nonstd::span records) override + { + if(records.empty()) + { + return; + } + + // Find source cell index range and face index range + usize minSrc = std::numeric_limits::max(); + usize maxSrc = 0; + usize minFace = std::numeric_limits::max(); + usize maxFace = 0; + for(const auto& r : records) + { + if(r.faceLabel0 != -1) + { + minSrc = std::min(minSrc, r.firstcIndex); + maxSrc = std::max(maxSrc, r.firstcIndex); + } + if(r.faceLabel1 != -1) + { + minSrc = std::min(minSrc, r.secondcIndex); + maxSrc = std::max(maxSrc, r.secondcIndex); + } + minFace = std::min(minFace, r.faceIndex); + maxFace = std::max(maxFace, r.faceIndex); + } + + if(minSrc > maxSrc) + { + return; // all exterior faces, nothing to copy from source + } + + // Bulk read source cell data + usize srcTupleCount = maxSrc - minSrc + 1; + auto srcBuf = std::make_unique(srcTupleCount * m_NumComps); + m_CellRef.copyIntoBuffer(minSrc * m_NumComps, nonstd::span(srcBuf.get(), srcTupleCount * m_NumComps)); + + // Build destination buffer + usize faceCount = maxFace - minFace + 1; + auto destBuf = std::make_unique(faceCount * m_NumComps * 2); + std::fill_n(destBuf.get(), faceCount * m_NumComps * 2, T{}); + + // Process all records using local buffers + for(const auto& r : records) + { + usize localFace = r.faceIndex - minFace; + if(r.faceLabel0 != -1) + { + usize srcOff = (r.firstcIndex - minSrc) * m_NumComps; + usize destOff = localFace * m_NumComps * 2; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = srcBuf[srcOff + c]; + } + } + if(r.faceLabel1 != -1) + { + usize srcOff = (r.secondcIndex - minSrc) * m_NumComps; + usize destOff = localFace * m_NumComps * 2 + m_NumComps; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = srcBuf[srcOff + c]; + } + } + } + + // Bulk write destination face data + m_FaceRef.copyFromBuffer(minFace * m_NumComps * 2, nonstd::span(destBuf.get(), faceCount * m_NumComps * 2)); + } + + + void surfaceNetsTransferBatch(nonstd::span records) override + { + if(records.empty()) + { + return; + } + + constexpr usize k_MaxIdx = std::numeric_limits::max(); + + usize minSrc = k_MaxIdx; + usize maxSrc = 0; + usize minFace = k_MaxIdx; + usize maxFace = 0; + for(const auto& r : records) + { + if(r.quadNxArrayIndices[0] != k_MaxIdx) + { + minSrc = std::min(minSrc, r.quadNxArrayIndices[0]); + maxSrc = std::max(maxSrc, r.quadNxArrayIndices[0]); + } + if(r.quadNxArrayIndices[1] != k_MaxIdx) + { + minSrc = std::min(minSrc, r.quadNxArrayIndices[1]); + maxSrc = std::max(maxSrc, r.quadNxArrayIndices[1]); + } + minFace = std::min(minFace, r.faceIndex); + maxFace = std::max(maxFace, r.faceIndex); + } + + if(minSrc > maxSrc) + { + return; + } + + usize srcTupleCount = maxSrc - minSrc + 1; + auto srcBuf = std::make_unique(srcTupleCount * m_NumComps); + m_CellRef.copyIntoBuffer(minSrc * m_NumComps, nonstd::span(srcBuf.get(), srcTupleCount * m_NumComps)); + + usize faceCount = maxFace - minFace + 1; + auto destBuf = std::make_unique(faceCount * m_NumComps * 2); + std::fill_n(destBuf.get(), faceCount * m_NumComps * 2, T{}); + + for(const auto& r : records) + { + usize localFace = r.faceIndex - minFace; + if(r.quadNxArrayIndices[0] != k_MaxIdx) + { + usize srcOff = (r.quadNxArrayIndices[0] - minSrc) * m_NumComps; + usize destOff = localFace * m_NumComps * 2; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = srcBuf[srcOff + c]; + } + } + if(r.quadNxArrayIndices[1] != k_MaxIdx) + { + usize srcOff = (r.quadNxArrayIndices[1] - minSrc) * m_NumComps; + usize destOff = localFace * m_NumComps * 2 + m_NumComps; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = srcBuf[srcOff + c]; + } + } + } + + m_FaceRef.copyFromBuffer(minFace * m_NumComps * 2, nonstd::span(destBuf.get(), faceCount * m_NumComps * 2)); + } + private: DataStoreType& m_CellRef; DataStoreType& m_FaceRef; @@ -268,6 +432,159 @@ class TransferFeatureTuple : public AbstractTupleTransfer } } + void quickSurfaceTransferBatch(nonstd::span records) override + { + if(records.empty()) + { + return; + } + + // Find cell index range and face index range + usize minSrc = std::numeric_limits::max(); + usize maxSrc = 0; + usize minFace = std::numeric_limits::max(); + usize maxFace = 0; + for(const auto& r : records) + { + if(r.faceLabel0 != -1) + { + minSrc = std::min(minSrc, r.firstcIndex); + maxSrc = std::max(maxSrc, r.firstcIndex); + } + if(r.faceLabel1 != -1) + { + minSrc = std::min(minSrc, r.secondcIndex); + maxSrc = std::max(maxSrc, r.secondcIndex); + } + minFace = std::min(minFace, r.faceIndex); + maxFace = std::max(maxFace, r.faceIndex); + } + + if(minSrc > maxSrc) + { + return; + } + + // Bulk read featureIds for the cell range + usize srcTupleCount = maxSrc - minSrc + 1; + auto featureIdBuf = std::make_unique(srcTupleCount); + m_FeatureIdsRef.copyIntoBuffer(minSrc, nonstd::span(featureIdBuf.get(), srcTupleCount)); + + // Feature data is small (feature-level, not cell-level) — cache it all + usize featureTuples = m_FeatureDataRef.getNumberOfTuples(); + auto featureDataBuf = std::make_unique(featureTuples * m_NumComps); + m_FeatureDataRef.copyIntoBuffer(0, nonstd::span(featureDataBuf.get(), featureTuples * m_NumComps)); + + // Build destination buffer + usize faceCount = maxFace - minFace + 1; + auto destBuf = std::make_unique(faceCount * m_NumComps * 2); + std::fill_n(destBuf.get(), faceCount * m_NumComps * 2, T{}); + + // Process records + for(const auto& r : records) + { + usize localFace = r.faceIndex - minFace; + if(r.faceLabel0 != -1) + { + K featureId = featureIdBuf[r.firstcIndex - minSrc]; + usize srcOff = featureId * m_NumComps; + usize destOff = localFace * m_NumComps * 2; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = featureDataBuf[srcOff + c]; + } + } + if(r.faceLabel1 != -1) + { + K featureId = featureIdBuf[r.secondcIndex - minSrc]; + usize srcOff = featureId * m_NumComps; + usize destOff = localFace * m_NumComps * 2 + m_NumComps; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = featureDataBuf[srcOff + c]; + } + } + } + + // Bulk write destination + m_FaceRef.copyFromBuffer(minFace * m_NumComps * 2, nonstd::span(destBuf.get(), faceCount * m_NumComps * 2)); + } + + + void surfaceNetsTransferBatch(nonstd::span records) override + { + if(records.empty()) + { + return; + } + + constexpr usize k_MaxIdx = std::numeric_limits::max(); + + usize minSrc = k_MaxIdx; + usize maxSrc = 0; + usize minFace = k_MaxIdx; + usize maxFace = 0; + for(const auto& r : records) + { + if(r.quadNxArrayIndices[0] != k_MaxIdx) + { + minSrc = std::min(minSrc, r.quadNxArrayIndices[0]); + maxSrc = std::max(maxSrc, r.quadNxArrayIndices[0]); + } + if(r.quadNxArrayIndices[1] != k_MaxIdx) + { + minSrc = std::min(minSrc, r.quadNxArrayIndices[1]); + maxSrc = std::max(maxSrc, r.quadNxArrayIndices[1]); + } + minFace = std::min(minFace, r.faceIndex); + maxFace = std::max(maxFace, r.faceIndex); + } + + if(minSrc > maxSrc) + { + return; + } + + usize srcTupleCount = maxSrc - minSrc + 1; + auto featureIdBuf = std::make_unique(srcTupleCount); + m_FeatureIdsRef.copyIntoBuffer(minSrc, nonstd::span(featureIdBuf.get(), srcTupleCount)); + + usize featureTuples = m_FeatureDataRef.getNumberOfTuples(); + auto featureDataBuf = std::make_unique(featureTuples * m_NumComps); + m_FeatureDataRef.copyIntoBuffer(0, nonstd::span(featureDataBuf.get(), featureTuples * m_NumComps)); + + usize faceCount = maxFace - minFace + 1; + auto destBuf = std::make_unique(faceCount * m_NumComps * 2); + std::fill_n(destBuf.get(), faceCount * m_NumComps * 2, T{}); + + for(const auto& r : records) + { + usize localFace = r.faceIndex - minFace; + if(r.quadNxArrayIndices[0] != k_MaxIdx) + { + K featureId = featureIdBuf[r.quadNxArrayIndices[0] - minSrc]; + usize srcOff = featureId * m_NumComps; + usize destOff = localFace * m_NumComps * 2; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = featureDataBuf[srcOff + c]; + } + } + if(r.quadNxArrayIndices[1] != k_MaxIdx) + { + K featureId = featureIdBuf[r.quadNxArrayIndices[1] - minSrc]; + usize srcOff = featureId * m_NumComps; + usize destOff = localFace * m_NumComps * 2 + m_NumComps; + for(usize c = 0; c < m_NumComps; c++) + { + destBuf[destOff + c] = featureDataBuf[srcOff + c]; + } + } + } + + m_FaceRef.copyFromBuffer(minFace * m_NumComps * 2, nonstd::span(destBuf.get(), faceCount * m_NumComps * 2)); + } + private: DataStoreType& m_FeatureDataRef; DataStoreType& m_FaceRef; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp index 9dd463ce38..c3aef001fa 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp @@ -84,29 +84,51 @@ Result<> WriteAvizoRectilinearCoordinate::writeData(FILE* outputFile) const fprintf(outputFile, "@1 # FeatureIds in z, y, x with X moving fastest, then Y, then Z\n"); - const auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->template getIDataStoreRefAs>(); + const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); const usize totalPoints = featureIds.getNumberOfTuples(); + constexpr usize k_ChunkSize = 65536; + std::vector buffer(k_ChunkSize); + const auto& featureIdsStore = featureIds.getDataStoreRef(); + if(m_InputValues->WriteBinaryFile) { - fwrite(featureIds.data(), sizeof(int32_t), totalPoints, outputFile); + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) + { + if(m_ShouldCancel) + { + return {}; + } + const usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(buffer.data(), count)); + fwrite(buffer.data(), sizeof(int32), count, outputFile); + } } else { // The "20 Items" is purely arbitrary and is put in to try and save some space in the ASCII file - int count = 0; - for(size_t i = 0; i < totalPoints; ++i) + int itemCount = 0; + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) { - fprintf(outputFile, "%d", featureIds[i]); - if(count < 20) + if(m_ShouldCancel) { - fprintf(outputFile, " "); - count++; + return {}; } - else + const usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(buffer.data(), count)); + for(usize i = 0; i < count; ++i) { - fprintf(outputFile, "\n"); - count = 0; + fprintf(outputFile, "%d", buffer[i]); + if(itemCount < 20) + { + fprintf(outputFile, " "); + itemCount++; + } + else + { + fprintf(outputFile, "\n"); + itemCount = 0; + } } } } @@ -118,12 +140,12 @@ Result<> WriteAvizoRectilinearCoordinate::writeData(FILE* outputFile) const { for(int d = 0; d < 3; ++d) { - std::vector coords(dims[d]); - for(size_t i = 0; i < dims[d]; ++i) + std::vector coords(dims[d]); + for(usize i = 0; i < dims[d]; ++i) { coords[i] = origin[d] + (res[d] * i); } - fwrite(reinterpret_cast(coords.data()), sizeof(char), sizeof(char) * sizeof(float) * dims[d], outputFile); + fwrite(reinterpret_cast(coords.data()), sizeof(char), sizeof(char) * sizeof(float32) * dims[d], outputFile); fprintf(outputFile, "\n"); } } @@ -131,7 +153,7 @@ Result<> WriteAvizoRectilinearCoordinate::writeData(FILE* outputFile) const { for(int d = 0; d < 3; ++d) { - for(size_t i = 0; i < dims[d]; ++i) + for(usize i = 0; i < dims[d]; ++i) { fprintf(outputFile, "%f ", origin[d] + (res[d] * i)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp index 0ac3f5d4f1..d5a64ddf2a 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp @@ -3,7 +3,6 @@ #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" -#include #include using namespace nx::core; @@ -87,29 +86,50 @@ Result<> WriteAvizoUniformCoordinate::writeData(FILE* outputFile) const { fprintf(outputFile, "@1\n"); - const auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->template getIDataStoreRefAs>(); + const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); const usize totalPoints = featureIds.getNumberOfTuples(); + constexpr usize k_ChunkSize = 65536; + std::vector buffer(k_ChunkSize); + const auto& featureIdsStore = featureIds.getDataStoreRef(); if(m_InputValues->WriteBinaryFile) { - fwrite(featureIds.data(), sizeof(int32), totalPoints, outputFile); + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) + { + if(m_ShouldCancel) + { + return {}; + } + const usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(buffer.data(), count)); + fwrite(buffer.data(), sizeof(int32), count, outputFile); + } } else { // The "20 Items" is purely arbitrary and is put in to try and save some space in the ASCII file - int count = 0; - for(size_t i = 0; i < totalPoints; ++i) + int itemCount = 0; + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) { - fprintf(outputFile, "%d", featureIds[i]); - if(count < 20) + if(m_ShouldCancel) { - fprintf(outputFile, " "); - count++; + return {}; } - else + const usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(buffer.data(), count)); + for(usize i = 0; i < count; ++i) { - fprintf(outputFile, "\n"); - count = 0; + fprintf(outputFile, "%d", buffer[i]); + if(itemCount < 20) + { + fprintf(outputFile, " "); + itemCount++; + } + else + { + fprintf(outputFile, "\n"); + itemCount = 0; + } } } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp index 9827ca8563..e18a2a6888 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeBoundaryCellsFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeBoundaryCellsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/DataPath.hpp" @@ -126,7 +126,7 @@ Result<> ComputeBoundaryCellsFilter::executeImpl(DataStructure& dataStructure, c inputValues.FeatureIdsArrayPath = filterArgs.value(k_FeatureIdsArrayPath_Key); inputValues.BoundaryCellsArrayName = inputValues.FeatureIdsArrayPath.replaceName(filterArgs.value(k_BoundaryCellsArrayName_Key)); - return ComputeBoundaryCellsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeBoundaryCells(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp index ceaaaae48c..b518d3d59d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeFeatureNeighborsFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeFeatureNeighborsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -189,7 +189,7 @@ Result<> ComputeFeatureNeighborsFilter::executeImpl(DataStructure& dataStructure inputValues.SharedSurfaceAreaListPath = inputValues.CellFeatureArrayPath.createChildPath(filterArgs.value(k_SharedSurfaceAreaName_Key)); inputValues.SurfaceFeaturesPath = inputValues.CellFeatureArrayPath.createChildPath(filterArgs.value(k_SurfaceFeaturesName_Key)); - return ComputeFeatureNeighborsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeFeatureNeighbors(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp index 70e3287dd0..556944f6c8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeKMedoidsFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeKMedoidsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -197,7 +197,7 @@ Result<> ComputeKMedoidsFilter::executeImpl(DataStructure& dataStructure, const inputValues.ClusteringArrayPath = filterArgs.value(k_SelectedArrayPath_Key); inputValues.FeatureIdsArrayPath = inputValues.ClusteringArrayPath.replaceName(filterArgs.value(k_FeatureIdsArrayName_Key)); - return ComputeKMedoidsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeKMedoids(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp index 76d39de478..6da89db6e4 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeSurfaceAreaToVolumeFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -147,7 +147,7 @@ Result<> ComputeSurfaceAreaToVolumeFilter::executeImpl(DataStructure& dataStruct inputValues.SphericityArrayName = inputValues.NumCellsArrayPath.replaceName(filterArgs.value(k_SphericityArrayName_Key)); inputValues.InputImageGeometry = filterArgs.value(k_SelectedImageGeometryPath_Key); - return ComputeSurfaceAreaToVolumeDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeSurfaceAreaToVolume(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp index c2f4cc1672..f9808f0c63 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ComputeSurfaceFeaturesFilter.cpp @@ -1,6 +1,6 @@ #include "ComputeSurfaceFeaturesFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" @@ -126,7 +126,7 @@ Result<> ComputeSurfaceFeaturesFilter::executeImpl(DataStructure& dataStructure, inputValues.InputImageGeometryPath = filterArgs.value(k_FeatureGeometryPath_Key); inputValues.MarkFeature0Neighbors = filterArgs.value(k_MarkFeature0Neighbors); inputValues.SurfaceFeaturesArrayName = filterArgs.value(k_SurfaceFeaturesArrayName_Key); - return ComputeSurfaceFeaturesDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return ComputeSurfaceFeatures(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp index 2ce7bc136b..c378d5a743 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/DBSCANFilter.cpp @@ -1,6 +1,6 @@ #include "DBSCANFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/DBSCAN.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -69,7 +69,7 @@ Parameters DBSCANFilter::parameters() const // Create the parameter descriptors that are needed for this filter params.insertSeparator(Parameters::Separator{"Random Number Seed Parameters"}); params.insertLinkableParameter(std::make_unique( - k_ParseOrderIndex_Key, "Parse Order", "Whether to use random or low density first for parse order. See Documentation for further detail", to_underlying(DBSCANDirect::ParseOrder::LowDensityFirst), + k_ParseOrderIndex_Key, "Parse Order", "Whether to use random or low density first for parse order. See Documentation for further detail", to_underlying(DBSCAN::ParseOrder::LowDensityFirst), ChoicesParameter::Choices{"Low Density First", "Random", "Seeded Random"})); // sequence dependent DO NOT REORDER params.insert(std::make_unique>(k_SeedValue_Key, "Seed Value", "The seed fed into the random generator", std::mt19937::default_seed)); params.insert(std::make_unique(k_SeedArrayName_Key, "Stored Seed Value Array Name", "Name of array holding the seed value", "DBSCAN SeedValue")); @@ -100,9 +100,9 @@ Parameters DBSCANFilter::parameters() const std::make_unique(k_FeatureAMPath_Key, "Cluster Attribute Matrix", "The complete path to the attribute matrix in which to store to hold Cluster Data", DataPath{})); // Associate the Linkable Parameter(s) to the children parameters that they control - params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCANDirect::ParseOrder::Random))); - params.linkParameters(k_ParseOrderIndex_Key, k_SeedValue_Key, static_cast(to_underlying(DBSCANDirect::ParseOrder::SeededRandom))); - params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCANDirect::ParseOrder::SeededRandom))); + params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCAN::ParseOrder::Random))); + params.linkParameters(k_ParseOrderIndex_Key, k_SeedValue_Key, static_cast(to_underlying(DBSCAN::ParseOrder::SeededRandom))); + params.linkParameters(k_ParseOrderIndex_Key, k_SeedArrayName_Key, static_cast(to_underlying(DBSCAN::ParseOrder::SeededRandom))); params.linkParameters(k_UseMask_Key, k_MaskArrayPath_Key, true); return params; @@ -183,7 +183,7 @@ IFilter::PreflightResult DBSCANFilter::preflightImpl(const DataStructure& dataSt } // For caching seed run to run - if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCANDirect::ParseOrder::LowDensityFirst) + if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCAN::ParseOrder::LowDensityFirst) { auto createAction = std::make_unique(DataType::uint64, std::vector{1}, std::vector{1}, DataPath({filterArgs.value(k_SeedArrayName_Key)})); resultOutputActions.value().appendAction(std::move(createAction)); @@ -204,12 +204,12 @@ Result<> DBSCANFilter::executeImpl(DataStructure& dataStructure, const Arguments } auto seed = filterArgs.value(k_SeedValue_Key); - if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCANDirect::ParseOrder::SeededRandom) + if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCAN::ParseOrder::SeededRandom) { seed = static_cast(std::chrono::steady_clock::now().time_since_epoch().count()); } - if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCANDirect::ParseOrder::LowDensityFirst) + if(static_cast(filterArgs.value(k_ParseOrderIndex_Key)) != DBSCAN::ParseOrder::LowDensityFirst) { // Store Seed Value in Top Level Array dataStructure.getDataRefAs(DataPath({filterArgs.value(k_SeedArrayName_Key)}))[0] = seed; @@ -227,7 +227,7 @@ Result<> DBSCANFilter::executeImpl(DataStructure& dataStructure, const Arguments inputValues.ParseOrder = filterArgs.value(k_ParseOrderIndex_Key); inputValues.Seed = filterArgs.value(k_SeedValue_Key); - return DBSCANDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return DBSCAN(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp index 90fbf068bd..fb5f642883 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/FillBadDataFilter.cpp @@ -1,6 +1,6 @@ #include "FillBadDataFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp" +#include "SimplnxCore/Filters/Algorithms/FillBadData.hpp" #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/DataPath.hpp" @@ -130,7 +130,7 @@ Result<> FillBadDataFilter::executeImpl(DataStructure& dataStructure, const Argu inputValues.ignoredDataArrayPaths = filterArgs.value(k_IgnoredDataArrayPaths_Key); inputValues.inputImageGeometry = filterArgs.value(k_SelectedImageGeometryPath_Key); - return FillBadDataBFS(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return FillBadData(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp index 86b63937ab..0c705e6b60 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/IdentifySampleFilter.cpp @@ -1,6 +1,6 @@ #include "IdentifySampleFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp" +#include "SimplnxCore/Filters/Algorithms/IdentifySample.hpp" #include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" @@ -110,7 +110,7 @@ Result<> IdentifySampleFilter::executeImpl(DataStructure& dataStructure, const A inputValues.InputImageGeometryPath = filterArgs.value(k_SelectedImageGeometryPath_Key); inputValues.MaskArrayPath = filterArgs.value(k_MaskArrayPath_Key); - return IdentifySampleBFS(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return IdentifySample(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp index 4627a932ac..1834d2237d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/MultiThresholdObjectsFilter.cpp @@ -1,6 +1,6 @@ #include "MultiThresholdObjectsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp" #include "simplnx/Common/TypeTraits.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -245,7 +245,7 @@ Result<> MultiThresholdObjectsFilter::executeImpl(DataStructure& dataStructure, inputValues.UseCustomFalseValue = filterArgs.value(k_UseCustomFalseValue); inputValues.UseCustomTrueValue = filterArgs.value(k_UseCustomTrueValue); - return MultiThresholdObjectsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return MultiThresholdObjects(dataStructure, messageHandler, shouldCancel, &inputValues)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp index 5e58eadc79..bc0fba90bc 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/QuickSurfaceMeshFilter.cpp @@ -1,6 +1,6 @@ #include "QuickSurfaceMeshFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/Filter/Actions/CopyArrayInstanceAction.hpp" @@ -235,7 +235,7 @@ Result<> QuickSurfaceMeshFilter::executeImpl(DataStructure& dataStructure, const } inputValues.CreatedDataArrayPaths = createdDataPaths; - return nx::core::QuickSurfaceMeshDirect(dataStructure, &inputValues, shouldCancel, messageHandler)(); + return nx::core::QuickSurfaceMesh(dataStructure, &inputValues, shouldCancel, messageHandler)(); } namespace diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp index 41f3525f54..c9fb9862b7 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/SurfaceNetsFilter.cpp @@ -1,6 +1,6 @@ #include "SurfaceNetsFilter.hpp" -#include "SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/SurfaceNets.hpp" #include "simplnx/DataStructure/DataPath.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" @@ -227,6 +227,6 @@ Result<> SurfaceNetsFilter::executeImpl(DataStructure& dataStructure, const Argu } inputValues.CreatedDataArrayPaths = createdDataPaths; - return SurfaceNetsDirect(dataStructure, messageHandler, shouldCancel, &inputValues)(); + return SurfaceNets(dataStructure, messageHandler, shouldCancel, &inputValues)(); } } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp index 9d96dfb7ae..6675afbf98 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/utils/VtkUtilities.hpp @@ -127,9 +127,8 @@ std::string TypeForPrimitive(const IFilter::MessageHandler& messageHandler) { messageHandler( IFilter::Message::Type::Info, - fmt::format( - "You are using 'long int' as a type which is not 32/64 bit safe. It is suggested you use one of the H5SupportTypes defined in such as int32_t or uint32_t.", - typeid(T).name())); + fmt::format("You are using 'long int' as a type which is not 32/64 bit safe. It is suggested you use one of the H5SupportTypes defined in such as int32 or uint32.", + typeid(T).name())); } return ""; } @@ -225,7 +224,7 @@ struct WriteVtkDataArrayFunctor { std::string buffer; buffer.reserve(k_BufferDumpVal); - for(size_t i = 0; i < totalElements; i++) + for(usize i = 0; i < totalElements; i++) { if(i % 20 == 0 && i > 0) { @@ -344,17 +343,17 @@ struct WriteVtkDataFunctor } else { - const size_t k_DefaultElementsPerLine = 10; + const usize k_DefaultElementsPerLine = 10; auto start = std::chrono::steady_clock::now(); auto numTuples = dataStoreRef.getSize(); - size_t currentItemCount = 0; + usize currentItemCount = 0; - for(size_t idx = 0; idx < numTuples; idx++) + for(usize idx = 0; idx < numTuples; idx++) { auto now = std::chrono::steady_clock::now(); if(std::chrono::duration_cast(now - start).count() > 1000) { - auto string = fmt::format("Processing {}: {}% completed", dataArrayRef.getName(), static_cast(100 * static_cast(idx) / static_cast(numTuples))); + auto string = fmt::format("Processing {}: {}% completed", dataArrayRef.getName(), static_cast(100 * static_cast(idx) / static_cast(numTuples))); messageHandler(IFilter::Message::Type::Info, string); start = now; if(shouldCancel) diff --git a/src/Plugins/SimplnxCore/test/AlignSectionsFeatureCentroidTest.cpp b/src/Plugins/SimplnxCore/test/AlignSectionsFeatureCentroidTest.cpp index 044d8cd011..a268988ba1 100644 --- a/src/Plugins/SimplnxCore/test/AlignSectionsFeatureCentroidTest.cpp +++ b/src/Plugins/SimplnxCore/test/AlignSectionsFeatureCentroidTest.cpp @@ -1,13 +1,16 @@ #include "SimplnxCore/Filters/AlignSectionsFeatureCentroidFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include -#include +#include #include namespace fs = std::filesystem; @@ -18,11 +21,19 @@ TEST_CASE("SimplnxCore::AlignSectionsFeatureCentroidFilter: Algorithm Test", "[R const DataPath k_ExemplarShiftsPath = Constants::k_ExemplarDataContainerPath.createChildPath("Exemplar Shifts"); UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_feature_centroids.tar.gz", "align_sections_feature_centroids"); // Read Exemplar DREAM3D File Filter auto exemplarFilePath = fs::path(fmt::format("{}/align_sections_feature_centroids/6_6_align_sections_feature_centroids.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(exemplarFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_MaskArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_MaskArrayPath)); // Align Sections Feature Centroid Filter { @@ -58,11 +69,20 @@ TEST_CASE("SimplnxCore::AlignSectionsFeatureCentroidFilter: output test", "[Reco { const std::string k_CentroidsName = "Centroids"; + UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_feature_centroids.tar.gz", "align_sections_feature_centroids"); // Read Exemplar DREAM3D File Filter auto baselineFilePath = fs::path(fmt::format("{}/align_sections_feature_centroids/6_6_align_sections_feature_centroids.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baselineFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_MaskArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_MaskArrayPath)); // Align Sections Feature Centroid Filter { @@ -98,15 +118,23 @@ TEST_CASE("SimplnxCore::AlignSectionsFeatureCentroidFilter: output test", "[Reco const DataPath alignmentAMPath = Constants::k_DataContainerPath.createChildPath(Constants::k_AlignmentAMName); const DataPath slicesPath = alignmentAMPath.createChildPath(Constants::k_SlicesArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(slicesPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(slicesPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(slicesPath), dataStructure.getDataRefAs(slicesPath)); const DataPath relativeShiftsPath = alignmentAMPath.createChildPath(Constants::k_RelativeShiftsArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(relativeShiftsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(relativeShiftsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(relativeShiftsPath), dataStructure.getDataRefAs(relativeShiftsPath)); const DataPath cumulativeShiftsPath = alignmentAMPath.createChildPath(Constants::k_CumulativeShiftsArrayName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(cumulativeShiftsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(cumulativeShiftsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(cumulativeShiftsPath), dataStructure.getDataRefAs(cumulativeShiftsPath)); const DataPath centroidsPath = alignmentAMPath.createChildPath(k_CentroidsName); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(centroidsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(centroidsPath)); UnitTest::CompareDataArrays(exemplarDataStructure.getDataRefAs(centroidsPath), dataStructure.getDataRefAs(centroidsPath)); // Write out the .dream3d file now @@ -116,3 +144,102 @@ TEST_CASE("SimplnxCore::AlignSectionsFeatureCentroidFilter: output test", "[Reco UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +TEST_CASE("SimplnxCore::AlignSectionsFeatureCentroid: Benchmark 200x200x200", "[SimplnxCore][AlignSectionsFeatureCentroidFilter][.Benchmark]") +{ + UnitTest::LoadPlugins(); + // 200x200x200, largest cell array is EulerAngles float32 3-comp => 200*200*3*4 = 480,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 480000, true); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + + constexpr usize kDimX = 200; + constexpr usize kDimY = 200; + constexpr usize kDimZ = 200; + constexpr usize kSliceVoxels = kDimX * kDimY; + const ShapeType cellTupleShape = {kDimZ, kDimY, kDimX}; + const auto benchmarkFile = fs::path(fmt::format("{}/align_sections_feature_centroid_benchmark.dream3d", unit_test::k_BinaryTestOutputDir)); + + // Stage 1: Build data programmatically and write to .dream3d + { + DataStructure buildDS; + auto* imageGeom = ImageGeom::Create(buildDS, "DataContainer"); + imageGeom->setDimensions({kDimX, kDimY, kDimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(buildDS, "CellData", cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto* maskArray = UnitTest::CreateTestDataArray(buildDS, "Mask", cellTupleShape, {1}, cellAM->getId()); + auto& maskStore = maskArray->getDataStoreRef(); + auto* eulerArray = UnitTest::CreateTestDataArray(buildDS, "EulerAngles", cellTupleShape, {3}, cellAM->getId()); + auto& eulerStore = eulerArray->getDataStoreRef(); + auto* featureIdsArray = UnitTest::CreateTestDataArray(buildDS, "FeatureIds", cellTupleShape, {1}, cellAM->getId()); + auto& featureIdsStore = featureIdsArray->getDataStoreRef(); + + // Fill using slice-at-a-time bulk writes + const float32 cx = kDimX / 2.0f; + const float32 cy = kDimY / 2.0f; + const float32 radius = 80.0f; + + std::vector maskBuf(kSliceVoxels); + std::vector eulerBuf(kSliceVoxels * 3); + std::vector featureIdsBuf(kSliceVoxels); + + for(usize z = 0; z < kDimZ; z++) + { + // Shift center per-slice to create meaningful alignment work + float32 wobbleX = 10.0f * std::sin(static_cast(z) * 0.1f); + float32 wobbleY = 8.0f * std::cos(static_cast(z) * 0.07f); + float32 sliceCx = cx + wobbleX; + float32 sliceCy = cy + wobbleY; + + for(usize y = 0; y < kDimY; y++) + { + for(usize x = 0; x < kDimX; x++) + { + const usize localIdx = y * kDimX + x; + const float32 dx = static_cast(x) - sliceCx; + const float32 dy = static_cast(y) - sliceCy; + const float32 dist = std::sqrt(dx * dx + dy * dy); + maskBuf[localIdx] = dist < radius ? 1 : 0; + + eulerBuf[localIdx * 3 + 0] = static_cast(x) * 0.01f; + eulerBuf[localIdx * 3 + 1] = static_cast(y) * 0.01f; + eulerBuf[localIdx * 3 + 2] = static_cast(z) * 0.01f; + featureIdsBuf[localIdx] = static_cast(maskBuf[localIdx]); + } + } + maskStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(maskBuf.data(), kSliceVoxels)); + eulerStore.copyFromBuffer(z * kSliceVoxels * 3, nonstd::span(eulerBuf.data(), kSliceVoxels * 3)); + featureIdsStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(featureIdsBuf.data(), kSliceVoxels)); + } + + UnitTest::WriteTestDataStructure(buildDS, benchmarkFile); + } + + // Stage 2: Reload (arrays become OOC-backed) and run filter + DataStructure dataStructure = UnitTest::LoadDataStructure(benchmarkFile); + + { + AlignSectionsFeatureCentroidFilter filter; + Arguments args; + + args.insertOrAssign(AlignSectionsFeatureCentroidFilter::k_UseReferenceSlice_Key, std::make_any(true)); + args.insertOrAssign(AlignSectionsFeatureCentroidFilter::k_ReferenceSlice_Key, std::make_any(0)); + args.insertOrAssign(AlignSectionsFeatureCentroidFilter::k_MaskArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Mask"}))); + args.insertOrAssign(AlignSectionsFeatureCentroidFilter::k_SelectedImageGeometryPath_Key, std::make_any(DataPath({"DataContainer"}))); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + + fs::remove(benchmarkFile); +} diff --git a/src/Plugins/SimplnxCore/test/AlignSectionsListTest.cpp b/src/Plugins/SimplnxCore/test/AlignSectionsListTest.cpp index 04c1a6d915..9d5fdbaf51 100644 --- a/src/Plugins/SimplnxCore/test/AlignSectionsListTest.cpp +++ b/src/Plugins/SimplnxCore/test/AlignSectionsListTest.cpp @@ -1,13 +1,17 @@ #include "SimplnxCore/Filters/AlignSectionsListFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/DataArrayUtilities.hpp" #include "simplnx/Utilities/DataGroupUtilities.hpp" #include +#include #include namespace fs = std::filesystem; @@ -30,6 +34,11 @@ struct CompareArraysFunctor TEST_CASE("SimplnxCore::AlignSectionsListFilter: Relative Shifts execution", "[SimplnxCore][AlignSectionsListFilter]") { UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); auto app = Application::GetOrCreateInstance(); auto* filterList = app->getFilterList(); @@ -41,15 +50,19 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Relative Shifts execution", "[S // Read the Small IN100 Data set auto baseDataFilePath = fs::path(fmt::format("{}/Small_IN100.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); // Read Exemplar DREAM3D File Filter auto exemplarFilePath = fs::path(fmt::format("{}/align_sections_misorientation/output_align_sections_misorientation.dream3d", unit_test::k_TestFilesDir)); DataStructure exemplarDataStructure = UnitTest::LoadDataStructure(exemplarFilePath); const DataPath newShiftsPath = DataPath({Constants::k_RelativeShiftsArrayName}); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(k_AlignmentAMPath.createChildPath(Constants::k_RelativeShiftsArrayName))); auto& exemplarShifts = exemplarDataStructure.getDataRefAs(k_AlignmentAMPath.createChildPath(Constants::k_RelativeShiftsArrayName)); UnitTest::CreateTestDataArray(dataStructure, Constants::k_RelativeShiftsArrayName, exemplarShifts.getTupleShape(), exemplarShifts.getComponentShape()); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(newShiftsPath)); auto& newShifts = dataStructure.getDataRefAs(newShiftsPath); CopyFromArray::CopyData(exemplarShifts, newShifts, 0ULL, 0ULL, exemplarShifts.getNumberOfTuples()); @@ -77,6 +90,7 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Relative Shifts execution", "[S SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(nx::core::Constants::k_DataContainerPath)); const auto& imageGeom = dataStructure.getDataRefAs(nx::core::Constants::k_DataContainerPath); const auto& cellAttributeMatrix = imageGeom.getCellData(); std::optional> selectedCellArrays = GetAllChildDataPaths(dataStructure, nx::core::Constants::k_DataContainerPath.createChildPath(cellAttributeMatrix->getName())); @@ -84,6 +98,8 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Relative Shifts execution", "[S for(const auto& path : selectedCellArrays.value()) { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(path)); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(path)); const auto& computedIDataArray = dataStructure.getDataRefAs(path); const auto& exemplarIDataArray = exemplarDataStructure.getDataRefAs(path); @@ -100,6 +116,12 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Relative Shifts execution", "[S TEST_CASE("SimplnxCore::AlignSectionsListFilter: Cumulative Shifts execution", "[SimplnxCore][AlignSectionsListFilter]") { UnitTest::LoadPlugins(); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 600000, true); + + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + auto* filterList = Application::GetOrCreateInstance()->getFilterList(); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "align_sections_misorientation.tar.gz", "align_sections_misorientation"); @@ -109,15 +131,19 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Cumulative Shifts execution", " // Read the Small IN100 Data set auto baseDataFilePath = fs::path(fmt::format("{}/Small_IN100.dream3d", unit_test::k_TestFilesDir)); DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(Constants::k_PhasesArrayPath)); // Read Exemplar DREAM3D File Filter auto exemplarFilePath = fs::path(fmt::format("{}/align_sections_misorientation/output_align_sections_misorientation.dream3d", unit_test::k_TestFilesDir)); DataStructure exemplarDataStructure = UnitTest::LoadDataStructure(exemplarFilePath); const DataPath newShiftsPath = DataPath({Constants::k_CumulativeShiftsArrayName}); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(k_AlignmentAMPath.createChildPath(Constants::k_CumulativeShiftsArrayName))); auto& exemplarShifts = exemplarDataStructure.getDataRefAs(k_AlignmentAMPath.createChildPath(Constants::k_CumulativeShiftsArrayName)); UnitTest::CreateTestDataArray(dataStructure, Constants::k_CumulativeShiftsArrayName, exemplarShifts.getTupleShape(), exemplarShifts.getComponentShape()); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(newShiftsPath)); auto& newShifts = dataStructure.getDataRefAs(newShiftsPath); CopyFromArray::CopyData(exemplarShifts, newShifts, 0ULL, 0ULL, exemplarShifts.getNumberOfTuples()); @@ -145,6 +171,7 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Cumulative Shifts execution", " SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(nx::core::Constants::k_DataContainerPath)); const auto& imageGeom = dataStructure.getDataRefAs(nx::core::Constants::k_DataContainerPath); const auto& cellAttributeMatrix = imageGeom.getCellData(); std::optional> selectedCellArrays = GetAllChildDataPaths(dataStructure, nx::core::Constants::k_DataContainerPath.createChildPath(cellAttributeMatrix->getName())); @@ -152,6 +179,8 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Cumulative Shifts execution", " for(const auto& path : selectedCellArrays.value()) { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(path)); + REQUIRE_NOTHROW(exemplarDataStructure.getDataRefAs(path)); const auto& computedIDataArray = dataStructure.getDataRefAs(path); const auto& exemplarIDataArray = exemplarDataStructure.getDataRefAs(path); @@ -164,3 +193,97 @@ TEST_CASE("SimplnxCore::AlignSectionsListFilter: Cumulative Shifts execution", " UnitTest::CheckArraysInheritTupleDims(dataStructure, SmallIn100::k_TupleCheckIgnoredPaths); } + +TEST_CASE("SimplnxCore::AlignSectionsListFilter: Benchmark 200x200x200", "[SimplnxCore][AlignSectionsListFilter][.Benchmark]") +{ + UnitTest::LoadPlugins(); + // 200x200x200, largest cell array is EulerAngles float32 3-comp => 200*200*3*4 = 480,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 480000, true); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + + constexpr usize kDimX = 200; + constexpr usize kDimY = 200; + constexpr usize kDimZ = 200; + constexpr usize kSliceVoxels = kDimX * kDimY; + const ShapeType cellTupleShape = {kDimZ, kDimY, kDimX}; + const auto benchmarkFile = fs::path(fmt::format("{}/align_sections_list_benchmark.dream3d", unit_test::k_BinaryTestOutputDir)); + + // Stage 1: Build data programmatically and write to .dream3d + { + DataStructure buildDS; + auto* imageGeom = ImageGeom::Create(buildDS, "DataContainer"); + imageGeom->setDimensions({kDimX, kDimY, kDimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(buildDS, "CellData", cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto* eulerArray = UnitTest::CreateTestDataArray(buildDS, "EulerAngles", cellTupleShape, {3}, cellAM->getId()); + auto& eulerStore = eulerArray->getDataStoreRef(); + auto* featureIdsArray = UnitTest::CreateTestDataArray(buildDS, "FeatureIds", cellTupleShape, {1}, cellAM->getId()); + auto& featureIdsStore = featureIdsArray->getDataStoreRef(); + auto* maskArray = UnitTest::CreateTestDataArray(buildDS, "Mask", cellTupleShape, {1}, cellAM->getId()); + auto& maskStore = maskArray->getDataStoreRef(); + + // Fill using slice-at-a-time bulk writes + std::vector eulerBuf(kSliceVoxels * 3); + std::vector featureIdsBuf(kSliceVoxels); + std::vector maskBuf(kSliceVoxels); + + for(usize z = 0; z < kDimZ; z++) + { + for(usize y = 0; y < kDimY; y++) + { + for(usize x = 0; x < kDimX; x++) + { + const usize localIdx = y * kDimX + x; + eulerBuf[localIdx * 3 + 0] = static_cast(x) * 0.01f; + eulerBuf[localIdx * 3 + 1] = static_cast(y) * 0.01f; + eulerBuf[localIdx * 3 + 2] = static_cast(z) * 0.01f; + featureIdsBuf[localIdx] = static_cast((x / 25) * 64 + (y / 25) * 8 + (z / 25)); + maskBuf[localIdx] = 1; + } + } + eulerStore.copyFromBuffer(z * kSliceVoxels * 3, nonstd::span(eulerBuf.data(), kSliceVoxels * 3)); + featureIdsStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(featureIdsBuf.data(), kSliceVoxels)); + maskStore.copyFromBuffer(z * kSliceVoxels, nonstd::span(maskBuf.data(), kSliceVoxels)); + } + + // Create shifts array (int64, 2-comp) at top level with varying shifts + const ShapeType shiftsTupleShape = {kDimZ}; + auto* shiftsArray = UnitTest::CreateTestDataArray(buildDS, "RelativeShifts", shiftsTupleShape, {2}); + auto& shiftsStore = shiftsArray->getDataStoreRef(); + for(usize z = 0; z < kDimZ; z++) + { + shiftsStore[z * 2 + 0] = static_cast(3.0 * std::sin(static_cast(z) * 0.1)); + shiftsStore[z * 2 + 1] = static_cast(2.0 * std::cos(static_cast(z) * 0.07)); + } + + UnitTest::WriteTestDataStructure(buildDS, benchmarkFile); + } + + // Stage 2: Reload (arrays become OOC-backed) and run filter + DataStructure dataStructure = UnitTest::LoadDataStructure(benchmarkFile); + + { + AlignSectionsListFilter filter; + Arguments args; + + args.insertOrAssign(AlignSectionsListFilter::k_InputArrayType_Key, std::make_any(0ULL)); + args.insertOrAssign(AlignSectionsListFilter::k_SelectedImageGeometryPath_Key, std::make_any(DataPath({"DataContainer"}))); + args.insertOrAssign(AlignSectionsListFilter::k_ShiftsArrayPath_Key, std::make_any(DataPath({"RelativeShifts"}))); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + + fs::remove(benchmarkFile); +} diff --git a/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp b/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp index f5a9f4a393..76f665f675 100644 --- a/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp @@ -1,6 +1,11 @@ #include +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "SimplnxCore/Filters/ComputeBoundaryCellsFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" @@ -15,11 +20,54 @@ const DataPath k_GeometryPath({k_ExemplarDataContainer}); const DataPath k_FeatureIdsPath({k_ExemplarDataContainer, Constants::k_CellData, Constants::k_FeatureIds}); const DataPath k_ExemplarBoundaryCellsPath({k_ExemplarDataContainer, Constants::k_CellData, "BoundaryCellsWithBoundary"}); const DataPath k_ComputedBoundaryCellsPath({k_ExemplarDataContainer, Constants::k_CellData, k_ComputedBoundaryCellsName}); + +constexpr usize k_DimX = 200; +constexpr usize k_DimY = 200; +constexpr usize k_DimZ = 200; +constexpr usize k_TotalVoxels = k_DimX * k_DimY * k_DimZ; +const std::string k_GeomName = "ImageGeom"; + +void BuildOctantFeatureIds(DataStructure& ds) +{ + const ShapeType cellTupleShape = {k_DimZ, k_DimY, k_DimX}; + + auto* imageGeom = ImageGeom::Create(ds, k_GeomName); + imageGeom->setDimensions({k_DimX, k_DimY, k_DimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(ds, Constants::k_CellData, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto store = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* featureIds = DataArray::Create(ds, Constants::k_FeatureIds, store, cellAM->getId()); + auto& fidsRef = featureIds->getDataStoreRef(); + const usize sliceSize = k_DimY * k_DimX; + std::vector sliceBuffer(sliceSize); + for(usize iz = 0; iz < k_DimZ; iz++) + { + for(usize iy = 0; iy < k_DimY; iy++) + { + for(usize ix = 0; ix < k_DimX; ix++) + { + const usize inSlice = iy * k_DimX + ix; + const usize globalIdx = iz * sliceSize + inSlice; + const int32 octant = static_cast((iz >= k_DimZ / 2 ? 4 : 0) + (iy >= k_DimY / 2 ? 2 : 0) + (ix >= k_DimX / 2 ? 1 : 0)) + 1; + sliceBuffer[inSlice] = ((globalIdx * 7 + 13) % 100) < 5 ? 0 : octant; + } + } + fidsRef.copyFromBuffer(iz * sliceSize, nonstd::span(sliceBuffer.data(), sliceSize)); + } +} } // namespace TEST_CASE("SimplnxCore::ComputeBoundaryCellsFilter: Valid filter execution", "[ComputeBoundaryCellsFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 65536, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_find_boundary_cells.tar.gz", "6_6_FindBoundaryCellsExemplar.dream3d"); @@ -88,3 +136,64 @@ TEST_CASE("SimplnxCore::ComputeBoundaryCellsFilter: Invalid filter execution", " UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +TEST_CASE("SimplnxCore::ComputeBoundaryCellsFilter: Generate Large Test Dataset", "[SimplnxCore][ComputeBoundaryCellsFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); + + DataStructure buildDS; + BuildOctantFeatureIds(buildDS); + + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data"; + fs::create_directories(outputDir); + const auto outputFile = outputDir / "compute_boundary_cells_data.dream3d"; + UnitTest::WriteTestDataStructure(buildDS, outputFile); + fmt::print("Wrote test data to: {}\n", outputFile.string()); +} + +TEST_CASE("SimplnxCore::ComputeBoundaryCellsFilter: 200x200x200 octant features", "[SimplnxCore][ComputeBoundaryCellsFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 160000, true); + + DataStructure dataStructure; + BuildOctantFeatureIds(dataStructure); + + const DataPath benchGeomPath({k_GeomName}); + const DataPath benchFeatureIdsPath = benchGeomPath.createChildPath(Constants::k_CellData).createChildPath(Constants::k_FeatureIds); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(benchFeatureIdsPath)); + + { + ComputeBoundaryCellsFilter filter; + Arguments args; + args.insertOrAssign(ComputeBoundaryCellsFilter::k_IgnoreFeatureZero_Key, std::make_any(true)); + args.insertOrAssign(ComputeBoundaryCellsFilter::k_IncludeVolumeBoundary_Key, std::make_any(true)); + args.insertOrAssign(ComputeBoundaryCellsFilter::k_GeometryPath_Key, std::make_any(benchGeomPath)); + args.insertOrAssign(ComputeBoundaryCellsFilter::k_FeatureIdsArrayPath_Key, std::make_any(benchFeatureIdsPath)); + args.insertOrAssign(ComputeBoundaryCellsFilter::k_BoundaryCellsArrayName_Key, std::make_any("BoundaryCells")); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + // Verify output: BoundaryCells should have varied values + const DataPath boundaryCellsPath = benchGeomPath.createChildPath(Constants::k_CellData).createChildPath("BoundaryCells"); + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(boundaryCellsPath)); + const auto& boundaryCells = dataStructure.getDataRefAs>(boundaryCellsPath).getDataStoreRef(); + int8 minVal = boundaryCells[0]; + int8 maxVal = boundaryCells[0]; + for(usize i = 1; i < k_TotalVoxels; i++) + { + minVal = std::min(minVal, boundaryCells[i]); + maxVal = std::max(maxVal, boundaryCells[i]); + } + REQUIRE(minVal == 0); + REQUIRE(maxVal > 0); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} diff --git a/src/Plugins/SimplnxCore/test/ComputeFeatureNeighborsTest.cpp b/src/Plugins/SimplnxCore/test/ComputeFeatureNeighborsTest.cpp index 68f1638d6c..52920e4319 100644 --- a/src/Plugins/SimplnxCore/test/ComputeFeatureNeighborsTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeFeatureNeighborsTest.cpp @@ -2,6 +2,7 @@ #include "SimplnxCore/SimplnxCore_test_dirs.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include @@ -491,230 +492,492 @@ void ExecuteFilter(DataStructure& dataStructure, bool testBoundaryCells, bool te TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 0.0.0: Single Voxel - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = CreateSingleVoxelDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 0.0.1: Single Voxel - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = CreateSingleVoxelDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 0.0.2: Single Voxel - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = CreateSingleVoxelDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 0.0.3: Single Voxel - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = CreateSingleVoxelDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.0.0: 1D Z - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DZDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.0.1: 1D Z - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DZDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.0.2: 1D Z - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DZDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.0.3: 1D Z - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DZDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.1.0: 1D Y - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DYDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.1.1: 1D Y - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DYDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.1.2: 1D Y - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DYDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.1.3: 1D Y - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DYDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.2.0: 1D X - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DXDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.2.1: 1D X - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DXDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.2.2: 1D X - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DXDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 1.2.3: 1D X - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create1DXDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.0.0: 2D Empty Z - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyZDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.0.1: 2D Empty Z - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyZDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.0.2: 2D Empty Z - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyZDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.0.3: 2D Empty Z - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyZDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.1.0: 2D Empty Y - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyYDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.1.1: 2D Empty Y - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyYDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.1.2: 2D Empty Y - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyYDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.1.3: 2D Empty Y - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyYDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.2.0: 2D Empty X - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyXDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.2.1: 2D Empty X - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyXDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.2.2: 2D Empty X - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyXDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 2.2.3: 2D Empty X - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create2DEmptyXDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 3.0.0: 3D - Full Execution", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create3DDataStructure(); ExecuteFilter(dataStructure, true, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 3.0.1: 3D - No Boundary", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create3DDataStructure(); ExecuteFilter(dataStructure, false, true); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 3.0.2: 3D - No Surface Features", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create3DDataStructure(); ExecuteFilter(dataStructure, true, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Case 3.0.3: 3D - No Optionals", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + DataStructure dataStructure = Create3DDataStructure(); ExecuteFilter(dataStructure, false, false); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } TEST_CASE("SimplnxCore::ComputeFeatureNeighborsFilter: Legacy: SmallIn100", "[SimplnxCore][ComputeFeatureNeighborsFilter]") { + UnitTest::LoadPlugins(); + + // Test both algorithm paths (in-core + OOC) + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::TestFileSentinel testDataSentinel(unit_test::k_TestFilesDir, "6_6_stats_test_v2.tar.gz", "6_6_stats_test_v2.dream3d"); // Read the Small IN100 Data set auto baseDataFilePath = fs::path(fmt::format("{}/6_6_stats_test_v2.dream3d", unit_test::k_TestFilesDir)); diff --git a/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp b/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp index 110bcb34cc..9026379e54 100644 --- a/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp @@ -1,9 +1,14 @@ #include "SimplnxCore/Filters/ComputeSurfaceAreaToVolumeFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ArrayCreationParameter.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -21,11 +26,69 @@ const std::string k_SurfaceAreaVolumeRationArrayName("SurfaceAreaVolumeRatio"); const std::string k_SphericityArrayName("Sphericity"); const std::string k_SurfaceAreaVolumeRationArrayNameNX("SurfaceAreaVolumeRatioNX"); const std::string k_SphericityArrayNameNX("SphericityNX"); + +constexpr usize k_Dim = 200; +constexpr usize k_TotalVoxels = k_Dim * k_Dim * k_Dim; +constexpr int32 k_NumOctantFeatures = 8; +const std::string k_GeomName = "ImageGeom"; + +void BuildOctantWithNumCells(DataStructure& ds) +{ + const ShapeType cellTupleShape = {k_Dim, k_Dim, k_Dim}; + + auto* imageGeom = ImageGeom::Create(ds, k_GeomName); + imageGeom->setDimensions({k_Dim, k_Dim, k_Dim}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(ds, Constants::k_CellData, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto fidsStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* featureIds = DataArray::Create(ds, Constants::k_FeatureIds, fidsStore, cellAM->getId()); + auto& fidsRef = featureIds->getDataStoreRef(); + + std::vector featureCounts(k_NumOctantFeatures + 1, 0); + const usize sliceSize = k_Dim * k_Dim; + std::vector sliceBuffer(sliceSize); + for(usize iz = 0; iz < k_Dim; iz++) + { + for(usize iy = 0; iy < k_Dim; iy++) + { + for(usize ix = 0; ix < k_Dim; ix++) + { + const usize inSlice = iy * k_Dim + ix; + const usize globalIdx = iz * sliceSize + inSlice; + const int32 octant = static_cast((iz >= k_Dim / 2 ? 4 : 0) + (iy >= k_Dim / 2 ? 2 : 0) + (ix >= k_Dim / 2 ? 1 : 0)) + 1; + const int32 fid = ((globalIdx * 7 + 13) % 100) < 5 ? 0 : octant; + sliceBuffer[inSlice] = fid; + if(fid > 0) + { + featureCounts[fid]++; + } + } + } + fidsRef.copyFromBuffer(iz * sliceSize, nonstd::span(sliceBuffer.data(), sliceSize)); + } + + auto* cellFeatureAM = AttributeMatrix::Create(ds, Constants::k_CellFeatureData, {static_cast(k_NumOctantFeatures + 1)}, imageGeom->getId()); + + // NumCells is small (9 tuples) — per-element writes are fine + auto numCellsStore = DataStoreUtilities::CreateDataStore({static_cast(k_NumOctantFeatures + 1)}, {1}, IDataAction::Mode::Execute); + auto* numCells = DataArray::Create(ds, Constants::k_NumElements, numCellsStore, cellFeatureAM->getId()); + auto& numCellsRef = numCells->getDataStoreRef(); + std::vector localNumCells(featureCounts.begin(), featureCounts.end()); + numCellsRef.copyFromBuffer(0, nonstd::span(localNumCells.data(), localNumCells.size())); +} } // namespace TEST_CASE("SimplnxCore::ComputeSurfaceAreaToVolume", "[SimplnxCore][ComputeSurfaceAreaToVolume]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 25600, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_stats_test_v2.tar.gz", "6_6_stats_test_v2.dream3d"); @@ -64,9 +127,13 @@ TEST_CASE("SimplnxCore::ComputeSurfaceAreaToVolume", "[SimplnxCore][ComputeSurfa { DataPath exemplarPath({k_DataContainer, k_CellFeatureData, k_SurfaceAreaVolumeRationArrayName}); DataPath calculatedPath({k_DataContainer, k_CellFeatureData, k_SurfaceAreaVolumeRationArrayNameNX}); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(calculatedPath)); CompareDataArrays(dataStructure.getDataRefAs(exemplarPath), dataStructure.getDataRefAs(calculatedPath)); exemplarPath = DataPath({k_DataContainer, k_CellFeatureData, k_SphericityArrayName}); calculatedPath = DataPath({k_DataContainer, k_CellFeatureData, k_SphericityArrayNameNX}); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(calculatedPath)); CompareDataArrays(dataStructure.getDataRefAs(exemplarPath), dataStructure.getDataRefAs(calculatedPath)); } @@ -77,3 +144,68 @@ TEST_CASE("SimplnxCore::ComputeSurfaceAreaToVolume", "[SimplnxCore][ComputeSurfa UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +TEST_CASE("SimplnxCore::ComputeSurfaceAreaToVolumeFilter: Generate Large Test Dataset", "[SimplnxCore][ComputeSurfaceAreaToVolumeFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); + + DataStructure buildDS; + BuildOctantWithNumCells(buildDS); + + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data"; + fs::create_directories(outputDir); + const auto outputFile = outputDir / "compute_surface_area_to_volume_data.dream3d"; + UnitTest::WriteTestDataStructure(buildDS, outputFile); + fmt::print("Wrote test data to: {}\n", outputFile.string()); +} + +TEST_CASE("SimplnxCore::ComputeSurfaceAreaToVolumeFilter: 200x200x200 octant features", "[SimplnxCore][ComputeSurfaceAreaToVolumeFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 160000, true); + + DataStructure dataStructure; + BuildOctantWithNumCells(dataStructure); + + const DataPath benchGeomPath({k_GeomName}); + const DataPath benchFeatureIdsPath = benchGeomPath.createChildPath(Constants::k_CellData).createChildPath(Constants::k_FeatureIds); + const DataPath benchCellFeatureAMPath = benchGeomPath.createChildPath(Constants::k_CellFeatureData); + const DataPath benchNumCellsPath = benchCellFeatureAMPath.createChildPath(Constants::k_NumElements); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(benchFeatureIdsPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(benchFeatureIdsPath)); + + { + ComputeSurfaceAreaToVolumeFilter filter; + Arguments args; + args.insertOrAssign(ComputeSurfaceAreaToVolumeFilter::k_SelectedImageGeometryPath_Key, std::make_any(benchGeomPath)); + args.insertOrAssign(ComputeSurfaceAreaToVolumeFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(benchFeatureIdsPath)); + args.insertOrAssign(ComputeSurfaceAreaToVolumeFilter::k_NumCellsArrayPath_Key, std::make_any(benchNumCellsPath)); + args.insertOrAssign(ComputeSurfaceAreaToVolumeFilter::k_CalculateSphericity_Key, std::make_any(true)); + args.insertOrAssign(ComputeSurfaceAreaToVolumeFilter::k_SurfaceAreaVolumeRatioArrayName_Key, std::make_any("SurfaceAreaVolumeRatio")); + args.insertOrAssign(ComputeSurfaceAreaToVolumeFilter::k_SphericityArrayName_Key, std::make_any("Sphericity")); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + // Verify: SA/V ratios and sphericity should be positive for all features + const DataPath savPath = benchCellFeatureAMPath.createChildPath("SurfaceAreaVolumeRatio"); + const DataPath sphericityPath = benchCellFeatureAMPath.createChildPath("Sphericity"); + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(savPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(sphericityPath)); + const auto& savRatio = dataStructure.getDataRefAs>(savPath).getDataStoreRef(); + const auto& sphericity = dataStructure.getDataRefAs>(sphericityPath).getDataStoreRef(); + for(int32 f = 1; f <= k_NumOctantFeatures; f++) + { + REQUIRE(savRatio[f] > 0.0f); + REQUIRE(sphericity[f] > 0.0f); + REQUIRE(sphericity[f] <= 1.0f); + } + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} diff --git a/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp b/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp index e4dc938a41..aa0c906116 100644 --- a/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp @@ -3,8 +3,13 @@ #include "SimplnxCore/Filters/ReadRawBinaryFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/DynamicTableParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -66,6 +71,7 @@ void test_impl(const std::vector& geometryDims, const std::string& featu // Change Feature 470 to 0 REQUIRE(dataStructure.getDataAs(k_FeatureIDsPath) != nullptr); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIDsPath)); Int32Array& featureIds = dataStructure.getDataRefAs(k_FeatureIDsPath); std::replace(featureIds.begin(), featureIds.end(), 470, 0); @@ -119,11 +125,58 @@ void test_impl(const std::vector& geometryDims, const std::string& featu UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +constexpr usize k_BenchDim = 200; +constexpr usize k_BenchTotalVoxels = k_BenchDim * k_BenchDim * k_BenchDim; +constexpr usize k_BlockSize = 25; +constexpr usize k_BlocksPerDim = k_BenchDim / k_BlockSize; // 8 +constexpr int32 k_NumBlockFeatures = static_cast(k_BlocksPerDim * k_BlocksPerDim * k_BlocksPerDim); // 512 +const std::string k_BenchGeomName = "ImageGeom"; + +void BuildBlockFeatureIds(DataStructure& ds) +{ + const ShapeType cellTupleShape = {k_BenchDim, k_BenchDim, k_BenchDim}; + + auto* imageGeom = ImageGeom::Create(ds, k_BenchGeomName); + imageGeom->setDimensions({k_BenchDim, k_BenchDim, k_BenchDim}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(ds, Constants::k_CellData, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto store = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* featureIds = DataArray::Create(ds, Constants::k_FeatureIds, store, cellAM->getId()); + auto& fidsRef = featureIds->getDataStoreRef(); + const usize sliceSize = k_BenchDim * k_BenchDim; + std::vector sliceBuffer(sliceSize); + for(usize iz = 0; iz < k_BenchDim; iz++) + { + for(usize iy = 0; iy < k_BenchDim; iy++) + { + for(usize ix = 0; ix < k_BenchDim; ix++) + { + const usize bx = ix / k_BlockSize; + const usize by = iy / k_BlockSize; + const usize bz = iz / k_BlockSize; + const int32 blockId = static_cast(bz * k_BlocksPerDim * k_BlocksPerDim + by * k_BlocksPerDim + bx + 1); + sliceBuffer[iy * k_BenchDim + ix] = blockId; + } + } + fidsRef.copyFromBuffer(iz * sliceSize, nonstd::span(sliceBuffer.data(), sliceSize)); + } + + AttributeMatrix::Create(ds, Constants::k_CellFeatureData, {static_cast(k_NumBlockFeatures + 1)}, imageGeom->getId()); +} } // namespace TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 3D", "[SimplnxCore][ComputeSurfaceFeaturesFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 40000, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_test_data_1_v2.tar.gz", "6_5_test_data_1_v2"); // Read the Small IN100 Data set @@ -155,6 +208,7 @@ TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 3D", "[SimplnxCore][Comput REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedSurfaceFeaturesPath)); DataPath exemplaryDataPath = Constants::k_CellFeatureDataPath.createChildPath(k_SurfaceFeaturesExemplar); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplaryDataPath)); const BoolArray& featureArrayExemplary = dataStructure.getDataRefAs(exemplaryDataPath); const UInt8Array& createdFeatureArray = dataStructure.getDataRefAs(computedSurfaceFeaturesPath); @@ -173,6 +227,10 @@ TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 3D", "[SimplnxCore][Comput TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 2D(XY Plane)", "[SimplnxCore][ComputeSurfaceFeaturesFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 40000, true); test_impl(std::vector({100, 100, 1}), k_FeatureIds2DFileName, k_SurfaceFeatures2DExemplaryFileName); } @@ -180,6 +238,10 @@ TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 2D(XY Plane)", "[SimplnxCo TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 2D(XZ Plane)", "[SimplnxCore][ComputeSurfaceFeaturesFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 40000, true); test_impl(std::vector({100, 1, 100}), k_FeatureIds2DFileName, k_SurfaceFeatures2DExemplaryFileName); } @@ -187,6 +249,81 @@ TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 2D(XZ Plane)", "[SimplnxCo TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 2D(YZ Plane)", "[SimplnxCore][ComputeSurfaceFeaturesFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 40000, true); test_impl(std::vector({1, 100, 100}), k_FeatureIds2DFileName, k_SurfaceFeatures2DExemplaryFileName); } + +TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: Generate Large Test Dataset", "[SimplnxCore][ComputeSurfaceFeaturesFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); + + DataStructure buildDS; + BuildBlockFeatureIds(buildDS); + + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data"; + fs::create_directories(outputDir); + const auto outputFile = outputDir / "compute_surface_features_data.dream3d"; + UnitTest::WriteTestDataStructure(buildDS, outputFile); + fmt::print("Wrote test data to: {}\n", outputFile.string()); +} + +TEST_CASE("SimplnxCore::ComputeSurfaceFeaturesFilter: 200x200x200 block features", "[SimplnxCore][ComputeSurfaceFeaturesFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOoc = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOoc); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 160000, true); + + DataStructure dataStructure; + BuildBlockFeatureIds(dataStructure); + + const DataPath benchGeomPath({k_BenchGeomName}); + const DataPath benchFeatureIdsPath = benchGeomPath.createChildPath(Constants::k_CellData).createChildPath(Constants::k_FeatureIds); + const DataPath benchCellFeatureAMPath = benchGeomPath.createChildPath(Constants::k_CellFeatureData); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(benchFeatureIdsPath)); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(benchFeatureIdsPath)); + + { + ComputeSurfaceFeaturesFilter filter; + Arguments args; + args.insertOrAssign(ComputeSurfaceFeaturesFilter::k_FeatureGeometryPath_Key, std::make_any(benchGeomPath)); + args.insertOrAssign(ComputeSurfaceFeaturesFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(benchFeatureIdsPath)); + args.insertOrAssign(ComputeSurfaceFeaturesFilter::k_CellFeatureAttributeMatrixPath_Key, std::make_any(benchCellFeatureAMPath)); + args.insertOrAssign(ComputeSurfaceFeaturesFilter::k_SurfaceFeaturesArrayName_Key, std::make_any("SurfaceFeatures")); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + // Verify: 512 features, 296 surface + 216 interior + const DataPath surfaceFeaturesPath = benchCellFeatureAMPath.createChildPath("SurfaceFeatures"); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(surfaceFeaturesPath)); + const auto& surfaceFeatures = dataStructure.getDataRefAs(surfaceFeaturesPath).getDataStoreRef(); + usize surfaceCount = 0; + usize interiorCount = 0; + for(usize f = 1; f <= static_cast(k_NumBlockFeatures); f++) + { + if(surfaceFeatures[f]) + { + surfaceCount++; + } + else + { + interiorCount++; + } + } + REQUIRE(surfaceCount > 0); + REQUIRE(interiorCount > 0); + // 8x8x8 grid: 6x6x6=216 interior, 512-216=296 surface + REQUIRE(interiorCount == 216); + REQUIRE(surfaceCount == 296); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} diff --git a/src/Plugins/SimplnxCore/test/DBSCANTest.cpp b/src/Plugins/SimplnxCore/test/DBSCANTest.cpp index 292270de70..6d68ff413f 100644 --- a/src/Plugins/SimplnxCore/test/DBSCANTest.cpp +++ b/src/Plugins/SimplnxCore/test/DBSCANTest.cpp @@ -4,7 +4,7 @@ #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" -#include "SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp" +#include "SimplnxCore/Filters/Algorithms/DBSCAN.hpp" #include "SimplnxCore/Filters/DBSCANFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" @@ -62,7 +62,7 @@ void LDFTestCase2D(const DataPath& targetPath, float32 epsilonVal, int32 minPtsV Arguments args; // Create default Parameters for the filter. - args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCANDirect::ParseOrder::LowDensityFirst))); + args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCAN::ParseOrder::LowDensityFirst))); args.insertOrAssign(DBSCANFilter::k_Epsilon_Key, std::make_any(epsilonVal)); args.insertOrAssign(DBSCANFilter::k_MinPoints_Key, std::make_any(minPtsVal)); args.insertOrAssign(DBSCANFilter::k_UseMask_Key, std::make_any(false)); @@ -109,7 +109,7 @@ std::vector BinPoints(const Int32Array& dataArray) void RandomTestCase2D(const DataPath& targetPath, float32 epsilonVal, int32 minPtsVal, const DataPath& exemplarClusterIds, ChoicesParameter::ValueType randomType) { - REQUIRE((randomType == to_underlying(DBSCANDirect::ParseOrder::Random) || randomType == to_underlying(DBSCANDirect::ParseOrder::SeededRandom))); + REQUIRE((randomType == to_underlying(DBSCAN::ParseOrder::Random) || randomType == to_underlying(DBSCAN::ParseOrder::SeededRandom))); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "dbscan_test.tar.gz", "dbscan_test"); DataStructure dataStructure = UnitTest::LoadDataStructure(k_2DTestFile); @@ -127,7 +127,7 @@ void RandomTestCase2D(const DataPath& targetPath, float32 epsilonVal, int32 minP // Create default Parameters for the filter. args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(randomType)); - args.insertOrAssign(DBSCANFilter::k_SeedValue_Key, std::make_any(k_Seed)); // Will be ignored if randomType == DBSCANDirect::ParseOrder::Random + args.insertOrAssign(DBSCANFilter::k_SeedValue_Key, std::make_any(k_Seed)); // Will be ignored if randomType == DBSCAN::ParseOrder::Random args.insertOrAssign(DBSCANFilter::k_SeedArrayName_Key, std::make_any("seed_array")); args.insertOrAssign(DBSCANFilter::k_Epsilon_Key, std::make_any(epsilonVal)); args.insertOrAssign(DBSCANFilter::k_MinPoints_Key, std::make_any(minPtsVal)); @@ -195,8 +195,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Aniso", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 4; // The exemplars were generated with LDF ::LDFTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath); - ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCANDirect::ParseOrder::Random); - ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCAN::ParseOrder::Random); + ::RandomTestCase2D(k_AnisoArrayPath, epsVal, minPtsVal, k_AnsioClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Blobs", "[SimplnxCore][DBSCAN]") @@ -205,8 +205,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Blobs", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath); - ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCANDirect::ParseOrder::Random); - ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCAN::ParseOrder::Random); + ::RandomTestCase2D(k_BlobsArrayPath, epsVal, minPtsVal, k_BlobsClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Circles", "[SimplnxCore][DBSCAN]") @@ -215,8 +215,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Circles", "[SimplnxCore][DBSCAN]" int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath); - ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCANDirect::ParseOrder::Random); - ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCAN::ParseOrder::Random); + ::RandomTestCase2D(k_CirclesArrayPath, epsVal, minPtsVal, k_CirclesClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Moons", "[SimplnxCore][DBSCAN]") @@ -225,8 +225,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Noisy Moons", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath); - ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCANDirect::ParseOrder::Random); - ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCAN::ParseOrder::Random); + ::RandomTestCase2D(k_MoonsArrayPath, epsVal, minPtsVal, k_MoonsClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: No Structure", "[SimplnxCore][DBSCAN]") @@ -235,8 +235,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: No Structure", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath); - ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCANDirect::ParseOrder::Random); - ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCAN::ParseOrder::Random); + ::RandomTestCase2D(k_NoStructureArrayPath, epsVal, minPtsVal, k_NoStructureClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Varied", "[SimplnxCore][DBSCAN]") @@ -245,8 +245,8 @@ TEST_CASE("SimplnxCore::DBSCAN: 2D Test: Varied", "[SimplnxCore][DBSCAN]") int32 minPtsVal = 3; // The exemplars were generated with LDF ::LDFTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath); - ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCANDirect::ParseOrder::Random); - ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCANDirect::ParseOrder::SeededRandom); + ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCAN::ParseOrder::Random); + ::RandomTestCase2D(k_VariedArrayPath, epsVal, minPtsVal, k_VariedClusterArrayPath, DBSCAN::ParseOrder::SeededRandom); } TEST_CASE("SimplnxCore::DBSCAN: 3D Test (LowDensityFirst)", "[SimplnxCore][DBSCAN]") @@ -268,7 +268,7 @@ TEST_CASE("SimplnxCore::DBSCAN: 3D Test (LowDensityFirst)", "[SimplnxCore][DBSCA Arguments args; // Create default Parameters for the filter. - args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCANDirect::ParseOrder::LowDensityFirst))); + args.insertOrAssign(DBSCANFilter::k_ParseOrderIndex_Key, std::make_any(to_underlying(DBSCAN::ParseOrder::LowDensityFirst))); args.insertOrAssign(DBSCANFilter::k_Epsilon_Key, std::make_any(0.0099999998f)); args.insertOrAssign(DBSCANFilter::k_MinPoints_Key, std::make_any(5)); args.insertOrAssign(DBSCANFilter::k_UseMask_Key, std::make_any(false)); diff --git a/src/Plugins/SimplnxCore/test/ErodeDilateBadDataTest.cpp b/src/Plugins/SimplnxCore/test/ErodeDilateBadDataTest.cpp index a6d8be55e6..a7ed9f4a59 100644 --- a/src/Plugins/SimplnxCore/test/ErodeDilateBadDataTest.cpp +++ b/src/Plugins/SimplnxCore/test/ErodeDilateBadDataTest.cpp @@ -3,11 +3,14 @@ #include "SimplnxCore/Filters/ErodeDilateBadDataFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/Parameters/MultiArraySelectionParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" -#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -21,102 +24,209 @@ namespace constexpr ChoicesParameter::ValueType k_Dilate = 0ULL; constexpr ChoicesParameter::ValueType k_Erode = 1ULL; -const std::string k_EbsdScanDataName("EBSD Scan Data"); +const std::string k_GeomName("ImageGeom"); +const std::string k_CellDataName("CellData"); -const DataPath k_InputData({"Input Data"}); -const DataPath k_EbsdScanDataDataPath = k_InputData.createChildPath(k_EbsdScanDataName); -const DataPath k_FeatureIdsDataPath = k_EbsdScanDataDataPath.createChildPath("FeatureIds"); +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_CellDataPath = k_GeomPath.createChildPath(k_CellDataName); +const DataPath k_FeatureIdsPath = k_CellDataPath.createChildPath("FeatureIds"); -} // namespace - -TEST_CASE("SimplnxCore::ErodeDilateBadDataFilter(Erode)", "[SimplnxCore][ErodeDilateBadDataFilter]") +void BuildTestData(DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ, usize blockSize) { - UnitTest::LoadPlugins(); + const ShapeType cellTupleShape = {dimZ, dimY, dimX}; + const usize sliceSize = dimX * dimY; - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_erode_dilate_test.tar.gz", "6_6_erode_dilate_test"); + auto* imageGeom = ImageGeom::Create(dataStructure, k_GeomName); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); - UnitTest::LoadPlugins(); - - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/6_6_erode_dilate_test/6_6_erode_dilate_bad_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = LoadDataStructure(exemplarFilePath); + auto* cellAM = AttributeMatrix::Create(dataStructure, k_CellDataName, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); - { - const ErodeDilateBadDataFilter filter; - Arguments args; - - // Create default Parameters for the filter. - args.insertOrAssign(ErodeDilateBadDataFilter::k_Operation_Key, std::make_any(k_Erode)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_NumIterations_Key, std::make_any(2)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_XDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_YDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_ZDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsDataPath)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); - args.insertOrAssign(ErodeDilateBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_InputData)); + auto featureIdsDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* featureIdsArray = DataArray::Create(dataStructure, "FeatureIds", featureIdsDataStore, cellAM->getId()); + auto& featureIdsStore = featureIdsArray->getDataStoreRef(); - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + auto eulerDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {3}, IDataAction::Mode::Execute); + auto* eulerArray = DataArray::Create(dataStructure, "EulerAngles", eulerDataStore, cellAM->getId()); + auto& eulerStore = eulerArray->getDataStoreRef(); - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) - } + auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* phasesArray = DataArray::Create(dataStructure, "Phases", phasesDataStore, cellAM->getId()); + auto& phasesStore = phasesArray->getDataStoreRef(); -// Write the DataStructure out to the file system -#ifdef SIMPLNX_WRITE_TEST_OUTPUT - WriteTestDataStructure(dataStructure, fs::path(fmt::format("{}/7_0_erode_dilate_bad_data.dream3d", unit_test::k_BinaryTestOutputDir))); -#endif + const usize blocksPerDimX = dimX / blockSize; + const usize blocksPerDimY = dimY / blockSize; - const std::string k_ExemplarDataContainerName("Exemplar Bad Data Erode"); - const DataPath k_ErodeCellAttributeMatrixDataPath = DataPath({k_ExemplarDataContainerName, "EBSD Scan Data"}); + // Build data using Z-slice buffered writes for OOC efficiency + std::vector featureIdsBuf(sliceSize); + std::vector eulerBuf(sliceSize * 3); + std::vector phasesBuf(sliceSize); - UnitTest::CompareExemplarToGeneratedData(dataStructure, dataStructure, k_EbsdScanDataDataPath, k_ExemplarDataContainerName); + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize inSlice = y * dimX + x; + phasesBuf[inSlice] = 1; + + usize bx = x / blockSize; + usize by = y / blockSize; + usize bz = z / blockSize; + int32 blockFeatureId = static_cast(bz * blocksPerDimY * blocksPerDimX + by * blocksPerDimX + bx + 1); + + bool isBad = ((x * 7 + y * 13 + z * 29) % 10 == 0); + featureIdsBuf[inSlice] = isBad ? 0 : blockFeatureId; + + const usize eIdx = inSlice * 3; + eulerBuf[eIdx] = static_cast(x) / static_cast(dimX); + eulerBuf[eIdx + 1] = static_cast(y) / static_cast(dimY); + eulerBuf[eIdx + 2] = static_cast(z) / static_cast(dimZ); + } + } + const usize zOffset = z * sliceSize; + featureIdsStore.copyFromBuffer(zOffset, nonstd::span(featureIdsBuf.data(), sliceSize)); + eulerStore.copyFromBuffer(zOffset * 3, nonstd::span(eulerBuf.data(), sliceSize * 3)); + phasesStore.copyFromBuffer(zOffset, nonstd::span(phasesBuf.data(), sliceSize)); + } +} - UnitTest::CheckArraysInheritTupleDims(dataStructure); +usize CountBadVoxels(const DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ) +{ + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath).getDataStoreRef(); + const usize sliceSize = dimX * dimY; + std::vector buf(sliceSize); + usize count = 0; + for(usize z = 0; z < dimZ; z++) + { + featureIds.copyIntoBuffer(z * sliceSize, nonstd::span(buf.data(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(buf[i] == 0) + { + count++; + } + } + } + return count; } +} // namespace -TEST_CASE("SimplnxCore::ErodeDilateBadDataFilter(Dilate)", "[SimplnxCore][ErodeDilateBadDataFilter]") +TEST_CASE("SimplnxCore::ErodeDilateBadDataFilter: Small Correctness", "[SimplnxCore][ErodeDilateBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 20x20x20, EulerAngles (float32, 3-comp) => 20*20*3*4 = 4,800 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 4800, true); + + auto operation = GENERATE(k_Erode, k_Dilate); + std::string operationName = (operation == k_Erode) ? "Erode" : "Dilate"; + DYNAMIC_SECTION("Operation: " << operationName << " forceOoc: " << forceOocAlgo) + { + DataStructure dataStructure; + BuildTestData(dataStructure, 20, 20, 20, 5); + + const usize badCountBefore = CountBadVoxels(dataStructure, 20, 20, 20); + REQUIRE(badCountBefore > 0); + + { + const ErodeDilateBadDataFilter filter; + Arguments args; + args.insertOrAssign(ErodeDilateBadDataFilter::k_Operation_Key, std::make_any(operation)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_NumIterations_Key, std::make_any(2)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_XDirOn_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_YDirOn_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_ZDirOn_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsPath)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); + args.insertOrAssign(ErodeDilateBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + const usize badCountAfter = CountBadVoxels(dataStructure, 20, 20, 20); + if(operation == k_Erode) + { + REQUIRE(badCountAfter < badCountBefore); + } + else + { + REQUIRE(badCountAfter > badCountBefore); + } + + // TODO: Add exemplar comparison after exemplar archive is published + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_erode_dilate_test.tar.gz", "6_6_erode_dilate_test"); +TEST_CASE("SimplnxCore::ErodeDilateBadDataFilter: Generate Test Data", "[SimplnxCore][ErodeDilateBadDataFilter][.GenerateTestData]") +{ + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data" / "erode_dilate_bad_data"; + fs::create_directories(outputDir); - const std::string k_ExemplarDataContainerName("Exemplar Bad Data Dilate"); - const DataPath k_DilateCellAttributeMatrixDataPath = DataPath({k_ExemplarDataContainerName, "EBSD Scan Data"}); + // Small input data (20x20x20, blockSize=5) + { + DataStructure buildDS; + BuildTestData(buildDS, 20, 20, 20, 5); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "small_input.dream3d"); + fmt::print("Generated small input: {}\n", (outputDir / "small_input.dream3d").string()); + } - UnitTest::LoadPlugins(); + // Large input data (200x200x200, blockSize=25) + { + DataStructure buildDS; + BuildTestData(buildDS, 200, 200, 200, 25); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "large_input.dream3d"); + fmt::print("Generated large input: {}\n", (outputDir / "large_input.dream3d").string()); + } +} - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/6_6_erode_dilate_test/6_6_erode_dilate_bad_data.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = LoadDataStructure(exemplarFilePath); +TEST_CASE("SimplnxCore::ErodeDilateBadDataFilter: 200x200x200 Large OOC", "[SimplnxCore][ErodeDilateBadDataFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 200x200x200, EulerAngles (float32, 3-comp) => 200*200*3*4 = 480,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 480000, true); + DYNAMIC_SECTION("forceOoc: " << forceOocAlgo) { - const ErodeDilateBadDataFilter filter; + DataStructure dataStructure; + BuildTestData(dataStructure, 200, 200, 200, 25); - Arguments args; + const usize badCountBefore = CountBadVoxels(dataStructure, 200, 200, 200); + REQUIRE(badCountBefore > 0); - // Create default Parameters for the filter. - args.insertOrAssign(ErodeDilateBadDataFilter::k_Operation_Key, std::make_any(k_Dilate)); + const ErodeDilateBadDataFilter filter; + Arguments args; + args.insertOrAssign(ErodeDilateBadDataFilter::k_Operation_Key, std::make_any(k_Erode)); args.insertOrAssign(ErodeDilateBadDataFilter::k_NumIterations_Key, std::make_any(2)); args.insertOrAssign(ErodeDilateBadDataFilter::k_XDirOn_Key, std::make_any(true)); args.insertOrAssign(ErodeDilateBadDataFilter::k_YDirOn_Key, std::make_any(true)); args.insertOrAssign(ErodeDilateBadDataFilter::k_ZDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsDataPath)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsPath)); args.insertOrAssign(ErodeDilateBadDataFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); - args.insertOrAssign(ErodeDilateBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_InputData)); + args.insertOrAssign(ErodeDilateBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) - } + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - UnitTest::CompareExemplarToGeneratedData(dataStructure, dataStructure, k_EbsdScanDataDataPath, k_ExemplarDataContainerName); + const usize badCountAfter = CountBadVoxels(dataStructure, 200, 200, 200); + REQUIRE(badCountAfter < badCountBefore); - UnitTest::CheckArraysInheritTupleDims(dataStructure); + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } } diff --git a/src/Plugins/SimplnxCore/test/ErodeDilateCoordinationNumberTest.cpp b/src/Plugins/SimplnxCore/test/ErodeDilateCoordinationNumberTest.cpp index a06b1609f6..44c5fafc48 100644 --- a/src/Plugins/SimplnxCore/test/ErodeDilateCoordinationNumberTest.cpp +++ b/src/Plugins/SimplnxCore/test/ErodeDilateCoordinationNumberTest.cpp @@ -3,10 +3,14 @@ #include "SimplnxCore/Filters/ErodeDilateCoordinationNumberFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/MultiArraySelectionParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -17,48 +21,186 @@ using namespace nx::core::UnitTest; namespace { -const std::string k_EbsdScanDataName("EBSD Scan Data"); +const std::string k_GeomName("ImageGeom"); +const std::string k_CellDataName("CellData"); -const DataPath k_InputData({"Input Data"}); -const DataPath k_EbsdScanDataDataPath = k_InputData.createChildPath(k_EbsdScanDataName); -const DataPath k_FeatureIdsDataPath = k_EbsdScanDataDataPath.createChildPath("FeatureIds"); +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_CellDataPath = k_GeomPath.createChildPath(k_CellDataName); +const DataPath k_FeatureIdsPath = k_CellDataPath.createChildPath("FeatureIds"); -const std::string k_ExemplarDataContainerName("Exemplar Coordination Number"); -const DataPath k_ErodeCellAttributeMatrixDataPath = DataPath({k_ExemplarDataContainerName, k_EbsdScanDataName}); +void BuildTestData(DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ, usize blockSize) +{ + const ShapeType cellTupleShape = {dimZ, dimY, dimX}; + const usize sliceSize = dimX * dimY; + + auto* imageGeom = ImageGeom::Create(dataStructure, k_GeomName); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(dataStructure, k_CellDataName, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto featureIdsDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* featureIdsArray = DataArray::Create(dataStructure, "FeatureIds", featureIdsDataStore, cellAM->getId()); + auto& featureIdsStore = featureIdsArray->getDataStoreRef(); + + auto eulerDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {3}, IDataAction::Mode::Execute); + auto* eulerArray = DataArray::Create(dataStructure, "EulerAngles", eulerDataStore, cellAM->getId()); + auto& eulerStore = eulerArray->getDataStoreRef(); + + const usize blocksPerDimX = dimX / blockSize; + const usize blocksPerDimY = dimY / blockSize; + + std::vector featureIdsBuf(sliceSize); + std::vector eulerBuf(sliceSize * 3); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize inSlice = y * dimX + x; + + usize bx = x / blockSize; + usize by = y / blockSize; + usize bz = z / blockSize; + int32 blockFeatureId = static_cast(bz * blocksPerDimY * blocksPerDimX + by * blocksPerDimX + bx + 1); + + bool isBad = ((x * 7 + y * 13 + z * 29) % 7 == 0); + featureIdsBuf[inSlice] = isBad ? 0 : blockFeatureId; + + const usize eIdx = inSlice * 3; + eulerBuf[eIdx] = static_cast(x) / static_cast(dimX); + eulerBuf[eIdx + 1] = static_cast(y) / static_cast(dimY); + eulerBuf[eIdx + 2] = static_cast(z) / static_cast(dimZ); + } + } + const usize zOffset = z * sliceSize; + featureIdsStore.copyFromBuffer(zOffset, nonstd::span(featureIdsBuf.data(), sliceSize)); + eulerStore.copyFromBuffer(zOffset * 3, nonstd::span(eulerBuf.data(), sliceSize * 3)); + } +} + +usize CountBadVoxels(const DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ) +{ + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath).getDataStoreRef(); + const usize sliceSize = dimX * dimY; + std::vector buf(sliceSize); + usize count = 0; + for(usize z = 0; z < dimZ; z++) + { + featureIds.copyIntoBuffer(z * sliceSize, nonstd::span(buf.data(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(buf[i] == 0) + { + count++; + } + } + } + return count; +} } // namespace -TEST_CASE("SimplnxCore::ErodeDilateCoordinationNumberFilter", "[SimplnxCore][ErodeDilateCoordinationNumberFilter]") +TEST_CASE("SimplnxCore::ErodeDilateCoordinationNumberFilter: Small Correctness", "[SimplnxCore][ErodeDilateCoordinationNumberFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 20x20x20, EulerAngles (float32, 3-comp) => 20*20*3*4 = 4,800 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 4800, true); + + DYNAMIC_SECTION("forceOoc: " << forceOocAlgo) + { + DataStructure dataStructure; + BuildTestData(dataStructure, 20, 20, 20, 5); + + const usize badCountBefore = CountBadVoxels(dataStructure, 20, 20, 20); + REQUIRE(badCountBefore > 0); + + { + const ErodeDilateCoordinationNumberFilter filter; + Arguments args; + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_CoordinationNumber_Key, std::make_any(4)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_Loop_Key, std::make_any(false)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsPath)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_erode_dilate_test.tar.gz", "6_6_erode_dilate_test"); + const usize badCountAfter = CountBadVoxels(dataStructure, 20, 20, 20); + REQUIRE(badCountAfter < badCountBefore); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/6_6_erode_dilate_test/6_6_erode_dilate_coordination_number.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = LoadDataStructure(exemplarFilePath); + // TODO: Add exemplar comparison after exemplar archive is published + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +TEST_CASE("SimplnxCore::ErodeDilateCoordinationNumberFilter: Generate Test Data", "[SimplnxCore][ErodeDilateCoordinationNumberFilter][.GenerateTestData]") +{ + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data" / "erode_dilate_coordination_number"; + fs::create_directories(outputDir); + + // Small input data (20x20x20, blockSize=5) + { + DataStructure buildDS; + BuildTestData(buildDS, 20, 20, 20, 5); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "small_input.dream3d"); + fmt::print("Generated small input: {}\n", (outputDir / "small_input.dream3d").string()); + } + + // Large input data (200x200x200, blockSize=25) + { + DataStructure buildDS; + BuildTestData(buildDS, 200, 200, 200, 25); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "large_input.dream3d"); + fmt::print("Generated large input: {}\n", (outputDir / "large_input.dream3d").string()); + } +} + +TEST_CASE("SimplnxCore::ErodeDilateCoordinationNumberFilter: 200x200x200 Large OOC", "[SimplnxCore][ErodeDilateCoordinationNumberFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 200x200x200, EulerAngles (float32, 3-comp) => 200*200*3*4 = 480,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 480000, true); + + DYNAMIC_SECTION("forceOoc: " << forceOocAlgo) { - // Instantiate the filter, a DataStructure object and an Arguments Object + DataStructure dataStructure; + BuildTestData(dataStructure, 200, 200, 200, 25); + + const usize badCountBefore = CountBadVoxels(dataStructure, 200, 200, 200); + REQUIRE(badCountBefore > 0); + const ErodeDilateCoordinationNumberFilter filter; Arguments args; - - // Create default Parameters for the filter. - args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_CoordinationNumber_Key, std::make_any(6)); - args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_Loop_Key, std::make_any(false)); - args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsDataPath)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_CoordinationNumber_Key, std::make_any(4)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_Loop_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsPath)); args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); - args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_InputData)); + args.insertOrAssign(ErodeDilateCoordinationNumberFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) - } + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - UnitTest::CompareExemplarToGeneratedData(dataStructure, dataStructure, k_EbsdScanDataDataPath, k_ExemplarDataContainerName); + const usize badCountAfter = CountBadVoxels(dataStructure, 200, 200, 200); + REQUIRE(badCountAfter < badCountBefore); - UnitTest::CheckArraysInheritTupleDims(dataStructure); + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } } diff --git a/src/Plugins/SimplnxCore/test/ErodeDilateMaskTest.cpp b/src/Plugins/SimplnxCore/test/ErodeDilateMaskTest.cpp index d6a3229b13..cf69577695 100644 --- a/src/Plugins/SimplnxCore/test/ErodeDilateMaskTest.cpp +++ b/src/Plugins/SimplnxCore/test/ErodeDilateMaskTest.cpp @@ -3,12 +3,17 @@ #include "SimplnxCore/Filters/ErodeDilateMaskFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ArraySelectionParameter.hpp" #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include +#include namespace fs = std::filesystem; using namespace nx::core; @@ -20,94 +25,180 @@ namespace constexpr ChoicesParameter::ValueType k_Dilate = 0ULL; constexpr ChoicesParameter::ValueType k_Erode = 1ULL; -const std::string k_EbsdScanDataName("EBSD Scan Data"); +const std::string k_GeomName("ImageGeom"); +const std::string k_CellDataName("CellData"); -const DataPath k_InputData({"Input Data"}); -const DataPath k_EbsdScanDataDataPath = k_InputData.createChildPath(k_EbsdScanDataName); -const DataPath k_MaskArrayDataPath = k_EbsdScanDataDataPath.createChildPath("Mask"); +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_CellDataPath = k_GeomPath.createChildPath(k_CellDataName); +const DataPath k_MaskPath = k_CellDataPath.createChildPath("Mask"); -} // namespace - -TEST_CASE("SimplnxCore::ErodeDilateMaskFilter(Dilate)", "[SimplnxCore][ErodeDilateMaskFilter]") +void BuildTestData(DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ) { - UnitTest::LoadPlugins(); + const ShapeType cellTupleShape = {dimZ, dimY, dimX}; + const usize sliceSize = dimX * dimY; - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_erode_dilate_test.tar.gz", "6_6_erode_dilate_test"); + auto* imageGeom = ImageGeom::Create(dataStructure, k_GeomName); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); - const std::string k_ExemplarDataContainerName("Exemplar Mask Dilate"); - const DataPath k_DilateCellAttributeMatrixDataPath = DataPath({k_ExemplarDataContainerName, "EBSD Scan Data"}); + auto* cellAM = AttributeMatrix::Create(dataStructure, k_CellDataName, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/6_6_erode_dilate_test/6_6_erode_dilate_mask.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = LoadDataStructure(exemplarFilePath); + auto maskDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* maskArray = DataArray::Create(dataStructure, "Mask", maskDataStore, cellAM->getId()); + auto& maskStore = maskArray->getDataStoreRef(); - // Instantiate the filter, a DataStructure object and an Arguments Object - { - const ErodeDilateMaskFilter filter; - Arguments args; + // Use Z-slice buffered writes for OOC efficiency (bool needs raw array, not vector) + auto maskBuf = std::make_unique(sliceSize); - // Create default Parameters for the filter. - args.insertOrAssign(ErodeDilateMaskFilter::k_Operation_Key, std::make_any(k_Dilate)); - args.insertOrAssign(ErodeDilateMaskFilter::k_NumIterations_Key, std::make_any(2)); - args.insertOrAssign(ErodeDilateMaskFilter::k_XDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateMaskFilter::k_YDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateMaskFilter::k_ZDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateMaskFilter::k_MaskArrayPath_Key, std::make_any(k_MaskArrayDataPath)); - args.insertOrAssign(ErodeDilateMaskFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_InputData)); - - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) - - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize inSlice = y * dimX + x; + maskBuf[inSlice] = ((x * 7 + y * 13 + z * 29) % 3 != 0); + } + } + maskStore.copyFromBuffer(z * sliceSize, nonstd::span(maskBuf.get(), sliceSize)); } +} - UnitTest::CompareExemplarToGeneratedData(dataStructure, dataStructure, k_EbsdScanDataDataPath, k_ExemplarDataContainerName); - - UnitTest::CheckArraysInheritTupleDims(dataStructure); +usize CountTrueVoxels(const DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ) +{ + const auto& mask = dataStructure.getDataRefAs(k_MaskPath).getDataStoreRef(); + const usize sliceSize = dimX * dimY; + auto buf = std::make_unique(sliceSize); + usize count = 0; + for(usize z = 0; z < dimZ; z++) + { + mask.copyIntoBuffer(z * sliceSize, nonstd::span(buf.get(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(buf[i]) + { + count++; + } + } + } + return count; } +} // namespace -TEST_CASE("SimplnxCore::ErodeDilateMaskFilter(Erode)", "[SimplnxCore][ErodeDilateMaskFilter]") +TEST_CASE("SimplnxCore::ErodeDilateMaskFilter: Small Correctness", "[SimplnxCore][ErodeDilateMaskFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 20x20x20, Mask (bool, 1-comp) => 20*20*1 = 400 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 400, true); + + auto operation = GENERATE(k_Erode, k_Dilate); + std::string operationName = (operation == k_Erode) ? "Erode" : "Dilate"; + DYNAMIC_SECTION("Operation: " << operationName << " forceOoc: " << forceOocAlgo) + { + DataStructure dataStructure; + BuildTestData(dataStructure, 20, 20, 20); + + const usize trueCountBefore = CountTrueVoxels(dataStructure, 20, 20, 20); + REQUIRE(trueCountBefore > 0); + REQUIRE(trueCountBefore < 20 * 20 * 20); + + { + const ErodeDilateMaskFilter filter; + Arguments args; + args.insertOrAssign(ErodeDilateMaskFilter::k_Operation_Key, std::make_any(operation)); + args.insertOrAssign(ErodeDilateMaskFilter::k_NumIterations_Key, std::make_any(2)); + args.insertOrAssign(ErodeDilateMaskFilter::k_XDirOn_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateMaskFilter::k_YDirOn_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateMaskFilter::k_ZDirOn_Key, std::make_any(true)); + args.insertOrAssign(ErodeDilateMaskFilter::k_MaskArrayPath_Key, std::make_any(k_MaskPath)); + args.insertOrAssign(ErodeDilateMaskFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + const usize trueCountAfter = CountTrueVoxels(dataStructure, 20, 20, 20); + if(operation == k_Erode) + { + REQUIRE(trueCountAfter < trueCountBefore); + } + else + { + REQUIRE(trueCountAfter > trueCountBefore); + } + + // TODO: Add exemplar comparison after exemplar archive is published + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_erode_dilate_test.tar.gz", "6_6_erode_dilate_test"); +TEST_CASE("SimplnxCore::ErodeDilateMaskFilter: Generate Test Data", "[SimplnxCore][ErodeDilateMaskFilter][.GenerateTestData]") +{ + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data" / "erode_dilate_mask"; + fs::create_directories(outputDir); - const std::string k_ExemplarDataContainerName("Exemplar Mask Erode"); - const DataPath k_ErodeCellAttributeMatrixDataPath = DataPath({k_ExemplarDataContainerName, "EBSD Scan Data"}); + // Small input data (20x20x20) + { + DataStructure buildDS; + BuildTestData(buildDS, 20, 20, 20); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "small_input.dream3d"); + fmt::print("Generated small input: {}\n", (outputDir / "small_input.dream3d").string()); + } - UnitTest::LoadPlugins(); + // Large input data (200x200x200) + { + DataStructure buildDS; + BuildTestData(buildDS, 200, 200, 200); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "large_input.dream3d"); + fmt::print("Generated large input: {}\n", (outputDir / "large_input.dream3d").string()); + } +} - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/6_6_erode_dilate_test/6_6_erode_dilate_mask.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = LoadDataStructure(exemplarFilePath); +TEST_CASE("SimplnxCore::ErodeDilateMaskFilter: 200x200x200 Large OOC", "[SimplnxCore][ErodeDilateMaskFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 200x200x200, Mask (bool, 1-comp) => 200*200*1 = 40,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 40000, true); - // Instantiate the filter, a DataStructure object and an Arguments Object + DYNAMIC_SECTION("forceOoc: " << forceOocAlgo) { + DataStructure dataStructure; + BuildTestData(dataStructure, 200, 200, 200); + + const usize trueCountBefore = CountTrueVoxels(dataStructure, 200, 200, 200); + REQUIRE(trueCountBefore > 0); + REQUIRE(trueCountBefore < 200 * 200 * 200); + const ErodeDilateMaskFilter filter; Arguments args; - - // Create default Parameters for the filter. args.insertOrAssign(ErodeDilateMaskFilter::k_Operation_Key, std::make_any(k_Erode)); args.insertOrAssign(ErodeDilateMaskFilter::k_NumIterations_Key, std::make_any(2)); args.insertOrAssign(ErodeDilateMaskFilter::k_XDirOn_Key, std::make_any(true)); args.insertOrAssign(ErodeDilateMaskFilter::k_YDirOn_Key, std::make_any(true)); args.insertOrAssign(ErodeDilateMaskFilter::k_ZDirOn_Key, std::make_any(true)); - args.insertOrAssign(ErodeDilateMaskFilter::k_MaskArrayPath_Key, std::make_any(k_MaskArrayDataPath)); - args.insertOrAssign(ErodeDilateMaskFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_InputData)); + args.insertOrAssign(ErodeDilateMaskFilter::k_MaskArrayPath_Key, std::make_any(k_MaskPath)); + args.insertOrAssign(ErodeDilateMaskFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) - } + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - UnitTest::CompareExemplarToGeneratedData(dataStructure, dataStructure, k_EbsdScanDataDataPath, k_ExemplarDataContainerName); + const usize trueCountAfter = CountTrueVoxels(dataStructure, 200, 200, 200); + REQUIRE(trueCountAfter < trueCountBefore); - UnitTest::CheckArraysInheritTupleDims(dataStructure); + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } } diff --git a/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp b/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp index fc47364d82..cb1b79b284 100644 --- a/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp +++ b/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp @@ -1,12 +1,15 @@ #include +#include +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/MultiArraySelectionParameter.hpp" -#include "simplnx/Pipeline/AbstractPipelineNode.hpp" -#include "simplnx/Pipeline/Pipeline.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include "SimplnxCore/Filters/FillBadDataFilter.hpp" -#include "SimplnxCore/Filters/ReadDREAM3DFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" #include @@ -16,58 +19,83 @@ using namespace nx::core; using namespace nx::core::Constants; using namespace nx::core::UnitTest; -TEST_CASE("SimplnxCore::FillBadData_SmallIN100", "[Core][FillBadDataFilter]") +namespace { - // Load the Simplnx Application instance and load the plugins - UnitTest::LoadPlugins(); - - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/6_5_exemplar.dream3d", unit_test::k_TestFilesDir)); - DataStructure exemplarDataStructure = UnitTest::LoadDataStructure(exemplarFilePath); +/** + * @brief Builds a FillBadData test dataset with block-patterned FeatureIds + * and ~10% scattered bad voxels (FeatureId=0) using a deterministic pattern. + */ +void BuildFillBadDataTestData(DataStructure& ds, usize dimX, usize dimY, usize dimZ, usize blockSize, bool addLargeDefect = false) +{ + const ShapeType cellShape = {dimZ, dimY, dimX}; + auto* imageGeom = ImageGeom::Create(ds, "DataContainer"); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); - // Read the Small IN100 Data set - auto baseDataFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/6_5_input.dream3d", unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + auto* cellAM = AttributeMatrix::Create(ds, "CellData", cellShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); - { - // Instantiate the filter, a DataStructure object and an Arguments Object - FillBadDataFilter filter; - Arguments args; + auto featureIdsDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* featureIdsArray = DataArray::Create(ds, "FeatureIds", featureIdsDataStore, cellAM->getId()); + auto& featureIdsStore = featureIdsArray->getDataStoreRef(); - // Create default Parameters for the filter. - args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(1000)); - args.insertOrAssign(FillBadDataFilter::k_StoreAsNewPhase_Key, std::make_any(false)); - args.insertOrAssign(FillBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(k_FeatureIdsArrayPath)); - args.insertOrAssign(FillBadDataFilter::k_CellPhasesArrayPath_Key, std::make_any(k_PhasesArrayPath)); + auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* phasesArray = DataArray::Create(ds, "Phases", phasesDataStore, cellAM->getId()); + auto& phasesStore = phasesArray->getDataStoreRef(); - args.insertOrAssign(FillBadDataFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); - args.insertOrAssign(FillBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_DataContainerPath)); + const usize blocksPerDim = dimX / blockSize; + const usize sliceSize = dimY * dimX; + std::vector featureIdsSliceBuffer(sliceSize); + std::vector phasesSliceBuffer(sliceSize, 1); - // Preflight the filter and check the result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) - - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); //, nullptr, IFilter::MessageHandler{[](const IFilter::Message& message) { fmt::print("{}\n", message.message); }}); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + usize bx = x / blockSize; + usize by = y / blockSize; + usize bz = z / blockSize; + int32 blockFeatureId = static_cast(bz * blocksPerDim * blocksPerDim + by * blocksPerDim + bx + 1); + + // Scatter bad voxels: ~10% of voxels become bad (FeatureId=0) + bool isBad = ((x * 7 + y * 13 + z * 29) % 10 == 0); + featureIdsSliceBuffer[y * dimX + x] = isBad ? 0 : blockFeatureId; + } + } + + // Add a contiguous large defect: entire z=dimZ/2 plane set to FeatureId=0 + if(addLargeDefect && z == dimZ / 2) + { + std::fill(featureIdsSliceBuffer.begin(), featureIdsSliceBuffer.end(), 0); + } + + featureIdsStore.copyFromBuffer(z * sliceSize, nonstd::span(featureIdsSliceBuffer.data(), sliceSize)); + phasesStore.copyFromBuffer(z * sliceSize, nonstd::span(phasesSliceBuffer.data(), sliceSize)); } - - UnitTest::CompareExemplarToGeneratedData(dataStructure, exemplarDataStructure, k_CellAttributeMatrix, k_DataContainer); - - // Write the DataStructure out to the file system - // #ifdef SIMPLNX_WRITE_TEST_OUTPUT - WriteTestDataStructure(dataStructure, fs::path(fmt::format("{}/7_0_fill_bad_data.dream3d", unit_test::k_BinaryTestOutputDir))); - // #endif - - UnitTest::CheckArraysInheritTupleDims(dataStructure); } +// Exemplar archive +const std::string k_ArchiveName = "fill_bad_data_exemplars.tar.gz"; +const std::string k_DataDirName = "fill_bad_data_exemplars"; +const fs::path k_DataDir = fs::path(unit_test::k_TestFilesDir.view()) / k_DataDirName; +const fs::path k_ExemplarFile = k_DataDir / "fill_bad_data.dream3d"; + +// Test dimensions for 200^3 tests +constexpr usize k_Dim = 200; +constexpr usize k_BlockSize = 25; +constexpr int32 k_MinDefectSize = 50; +} // namespace TEST_CASE("SimplnxCore::FillBadData::Test01_SingleSmallDefect", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 100, true); // 100 bytes - force very small arrays to OOC + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 100, true); // 100 bytes - force very small arrays to OOC const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -79,6 +107,8 @@ TEST_CASE("SimplnxCore::FillBadData::Test01_SingleSmallDefect", "[Core][FillBadD auto expectedFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/test_01_expected.dream3d", unit_test::k_TestFilesDir)); DataStructure expectedDataStructure = UnitTest::LoadDataStructure(expectedFilePath); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + // Run filter FillBadDataFilter filter; Arguments args; @@ -104,9 +134,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test01_SingleSmallDefect", "[Core][FillBadD TEST_CASE("SimplnxCore::FillBadData::Test02_SingleLargeDefect", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 100, true); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 100, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -118,6 +151,8 @@ TEST_CASE("SimplnxCore::FillBadData::Test02_SingleLargeDefect", "[Core][FillBadD auto expectedFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/test_02_expected.dream3d", unit_test::k_TestFilesDir)); DataStructure expectedDataStructure = UnitTest::LoadDataStructure(expectedFilePath); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + // Run filter FillBadDataFilter filter; Arguments args; @@ -143,9 +178,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test02_SingleLargeDefect", "[Core][FillBadD TEST_CASE("SimplnxCore::FillBadData::Test03_ThresholdBoundary", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 100, true); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 100, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -155,6 +193,8 @@ TEST_CASE("SimplnxCore::FillBadData::Test03_ThresholdBoundary", "[Core][FillBadD auto expectedFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/test_03_expected.dream3d", unit_test::k_TestFilesDir)); DataStructure expectedDataStructure = UnitTest::LoadDataStructure(expectedFilePath); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + FillBadDataFilter filter; Arguments args; args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(25)); @@ -177,9 +217,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test03_ThresholdBoundary", "[Core][FillBadD TEST_CASE("SimplnxCore::FillBadData::Test04_MultipleSmallDefects", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 500, true); // Slightly larger for 10x10x10 + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 500, true); // Slightly larger for 10x10x10 const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -189,6 +232,8 @@ TEST_CASE("SimplnxCore::FillBadData::Test04_MultipleSmallDefects", "[Core][FillB auto expectedFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/test_04_expected.dream3d", unit_test::k_TestFilesDir)); DataStructure expectedDataStructure = UnitTest::LoadDataStructure(expectedFilePath); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + FillBadDataFilter filter; Arguments args; args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(50)); @@ -211,9 +256,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test04_MultipleSmallDefects", "[Core][FillB TEST_CASE("SimplnxCore::FillBadData::Test05_MixedSmallAndLarge", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 500, true); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 500, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -223,6 +271,8 @@ TEST_CASE("SimplnxCore::FillBadData::Test05_MixedSmallAndLarge", "[Core][FillBad auto expectedFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/test_05_expected.dream3d", unit_test::k_TestFilesDir)); DataStructure expectedDataStructure = UnitTest::LoadDataStructure(expectedFilePath); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + FillBadDataFilter filter; Arguments args; args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(50)); @@ -245,9 +295,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test05_MixedSmallAndLarge", "[Core][FillBad TEST_CASE("SimplnxCore::FillBadData::Test06_SingleVoxelDefects", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 100, true); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 100, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -257,6 +310,8 @@ TEST_CASE("SimplnxCore::FillBadData::Test06_SingleVoxelDefects", "[Core][FillBad auto expectedFilePath = fs::path(fmt::format("{}/6_5_fill_bad_data/test_06_expected.dream3d", unit_test::k_TestFilesDir)); DataStructure expectedDataStructure = UnitTest::LoadDataStructure(expectedFilePath); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + FillBadDataFilter filter; Arguments args; args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(10)); @@ -279,9 +334,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test06_SingleVoxelDefects", "[Core][FillBad TEST_CASE("SimplnxCore::FillBadData::Test07_DefectsAtBoundaries", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 100, true); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 100, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -313,9 +371,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test07_DefectsAtBoundaries", "[Core][FillBa TEST_CASE("SimplnxCore::FillBadData::Test11_NeighborTieBreaking", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 50, true); // Very small for 3x3x3 + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 50, true); // Very small for 3x3x3 const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -352,9 +413,12 @@ TEST_CASE("SimplnxCore::FillBadData::Test11_NeighborTieBreaking", "[Core][FillBa TEST_CASE("SimplnxCore::FillBadData::Test13_StoreAsNewPhase", "[Core][FillBadDataFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); // Configure out-of-core settings (automatically restored on scope exit) - const UnitTest::PreferencesSentinel prefsSentinel("Zarr", 100, true); + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 100, true); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_fill_bad_data.tar.gz", "6_5_fill_bad_data"); @@ -387,3 +451,181 @@ TEST_CASE("SimplnxCore::FillBadData::Test13_StoreAsNewPhase", "[Core][FillBadDat UnitTest::CheckArraysInheritTupleDims(dataStructure); } + +TEST_CASE("SimplnxCore::FillBadData: 200x200x200 Correctness", "[Core][FillBadDataFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // int32 1-comp => 200*200*4 = 160,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 160000, true); + + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_ExemplarFile); + + std::string testName = GENERATE("NoNewPhase", "NewPhase"); + DYNAMIC_SECTION("Variant: " << testName) + { + const bool storeAsNewPhase = (testName == "NewPhase"); + + DataStructure dataStructure; + BuildFillBadDataTestData(dataStructure, k_Dim, k_Dim, k_Dim, k_BlockSize, true); + + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + + FillBadDataFilter filter; + Arguments args; + args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(k_MinDefectSize)); + args.insertOrAssign(FillBadDataFilter::k_StoreAsNewPhase_Key, std::make_any(storeAsNewPhase)); + args.insertOrAssign(FillBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + args.insertOrAssign(FillBadDataFilter::k_CellPhasesArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Phases"}))); + args.insertOrAssign(FillBadDataFilter::k_IgnoredDataArrayPaths_Key, std::make_any(MultiArraySelectionParameter::ValueType{})); + args.insertOrAssign(FillBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(DataPath({"DataContainer"}))); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // Compare against exemplar + const std::string exemplarGeomName = "DataContainer_" + testName + "_Exemplar"; + const DataPath exemplarFeatureIdsPath({exemplarGeomName, "CellData", "FeatureIds"}); + const DataPath exemplarPhasesPath({exemplarGeomName, "CellData", "Phases"}); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "Phases"}))); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarPhasesPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarPhasesPath), dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "Phases"}))); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +TEST_CASE("SimplnxCore::FillBadData: 200x200x200 Ignored Arrays", "[Core][FillBadDataFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // int32 1-comp => 200*200*4 = 160,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 160000, true); + + constexpr int32 k_Sentinel = -999; + + DataStructure dataStructure; + BuildFillBadDataTestData(dataStructure, k_Dim, k_Dim, k_Dim, k_BlockSize, false); + + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + + // Add an extra "IgnoredArray" filled with a sentinel value + auto& cellAM = dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData"})); + auto ignoredDataStore = DataStoreUtilities::CreateDataStore(cellAM.getShape(), {1}, IDataAction::Mode::Execute); + auto* ignoredArray = DataArray::Create(dataStructure, "IgnoredArray", ignoredDataStore, cellAM.getId()); + auto& ignoredStore = ignoredArray->getDataStoreRef(); + ignoredStore.fill(k_Sentinel); + + // Record bad-voxel count before fill using bulk reads + const auto& featureIdsBefore = dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"})); + const auto& fidsStore = featureIdsBefore.getDataStoreRef(); + const usize totalTuples = fidsStore.getNumberOfTuples(); + usize badCount = 0; + { + constexpr usize kChunk = 40000; + auto buf = std::make_unique(kChunk); + for(usize off = 0; off < totalTuples; off += kChunk) + { + const usize count = std::min(kChunk, totalTuples - off); + fidsStore.copyIntoBuffer(off, nonstd::span(buf.get(), count)); + for(usize i = 0; i < count; i++) + { + if(buf[i] == 0) + { + badCount++; + } + } + } + } + REQUIRE(badCount > 0); + + { + FillBadDataFilter filter; + Arguments args; + args.insertOrAssign(FillBadDataFilter::k_MinAllowedDefectSize_Key, std::make_any(50)); + args.insertOrAssign(FillBadDataFilter::k_StoreAsNewPhase_Key, std::make_any(false)); + args.insertOrAssign(FillBadDataFilter::k_CellFeatureIdsArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "FeatureIds"}))); + args.insertOrAssign(FillBadDataFilter::k_CellPhasesArrayPath_Key, std::make_any(DataPath({"DataContainer", "CellData", "Phases"}))); + args.insertOrAssign(FillBadDataFilter::k_SelectedImageGeometryPath_Key, std::make_any(DataPath({"DataContainer"}))); + + // Include the IgnoredArray in the ignored paths + MultiArraySelectionParameter::ValueType ignoredPaths = {DataPath({"DataContainer", "CellData", "IgnoredArray"})}; + args.insertOrAssign(FillBadDataFilter::k_IgnoredDataArrayPaths_Key, std::make_any(ignoredPaths)); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + // Verify: FeatureIds has no zeros (all scattered bad voxels filled) — bulk read + const auto& featureIdsAfter = dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "FeatureIds"})); + { + const auto& fidsAfterStore = featureIdsAfter.getDataStoreRef(); + constexpr usize kChunk = 40000; + auto buf = std::make_unique(kChunk); + for(usize off = 0; off < totalTuples; off += kChunk) + { + const usize count = std::min(kChunk, totalTuples - off); + fidsAfterStore.copyIntoBuffer(off, nonstd::span(buf.get(), count)); + for(usize i = 0; i < count; i++) + { + if(buf[i] == 0) + { + UNSCOPED_INFO(fmt::format("FeatureIds still zero at index {}", off + i)); + REQUIRE(false); + } + } + } + } + + // Verify: IgnoredArray is completely unchanged (sentinel at every voxel) — bulk read + const auto& ignoredAfter = dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData", "IgnoredArray"})); + { + const auto& ignoredAfterStore = ignoredAfter.getDataStoreRef(); + constexpr usize kChunk = 40000; + auto buf = std::make_unique(kChunk); + for(usize off = 0; off < totalTuples; off += kChunk) + { + const usize count = std::min(kChunk, totalTuples - off); + ignoredAfterStore.copyIntoBuffer(off, nonstd::span(buf.get(), count)); + for(usize i = 0; i < count; i++) + { + if(buf[i] != k_Sentinel) + { + UNSCOPED_INFO(fmt::format("IgnoredArray changed at index {}: expected {} got {}", off + i, k_Sentinel, buf[i])); + REQUIRE(false); + } + } + } + } + + UnitTest::CheckArraysInheritTupleDims(dataStructure); +} + +TEST_CASE("SimplnxCore::FillBadData: Generate Test Data", "[Core][FillBadDataFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); + + const auto outputDir = fs::path(fmt::format("{}/generated_test_data/fill_bad_data", unit_test::k_BinaryTestOutputDir)); + fs::create_directories(outputDir); + + // 200^3 input data with large defect (full z=k_Dim/2 plane) + { + DataStructure ds; + BuildFillBadDataTestData(ds, k_Dim, k_Dim, k_Dim, k_BlockSize, true); + UnitTest::WriteTestDataStructure(ds, outputDir / "input.dream3d"); + } +} diff --git a/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp b/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp index c4dc3dc0cb..81c00298f7 100644 --- a/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp +++ b/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp @@ -2,90 +2,201 @@ #include "SimplnxCore/Filters/IdentifySampleFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" -#include "simplnx/DataStructure/IDataArray.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include +#include + +#include using namespace nx::core; using namespace nx::core::UnitTest; namespace { -const DataPath k_ExemplarArrayPath = Constants::k_DataContainerPath.createChildPath(Constants::k_CellData).createChildPath("Mask Exemplar"); +// Exemplar archive +const std::string k_ArchiveName = "identify_sample_exemplars.tar.gz"; +const std::string k_DataDirName = "identify_sample_exemplars"; +const fs::path k_DataDir = fs::path(unit_test::k_TestFilesDir.view()) / k_DataDirName; +const fs::path k_ExemplarFile = k_DataDir / "identify_sample.dream3d"; + +// Geometry names +constexpr StringLiteral k_GeomName = "DataContainer"; +constexpr StringLiteral k_CellDataName = "CellData"; + +// Output array paths +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_MaskPath({k_GeomName, k_CellDataName, "Mask"}); + +// Test dimensions +constexpr usize k_Dim = 200; + +/** + * @brief Builds an IdentifySample test dataset: a sphere of "good" voxels + * with interior holes and exterior noise. + */ +void BuildIdentifySampleTestData(DataStructure& ds, usize dimX, usize dimY, usize dimZ, const std::string& geomName = "DataContainer") +{ + const ShapeType cellShape = {dimZ, dimY, dimX}; + auto* imageGeom = ImageGeom::Create(ds, geomName); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(ds, "CellData", cellShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto maskDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* maskArray = DataArray::Create(ds, "Mask", maskDataStore, cellAM->getId()); + auto& maskStore = maskArray->getDataStoreRef(); + + const float32 cx = dimX / 2.0f; + const float32 cy = dimY / 2.0f; + const float32 cz = dimZ / 2.0f; + const float32 radius = dimX * 0.4f; + + const usize sliceSize = dimY * dimX; + std::vector sliceBuffer(sliceSize); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const float32 dx = static_cast(x) - cx; + const float32 dy = static_cast(y) - cy; + const float32 dz = static_cast(z) - cz; + const float32 dist = std::sqrt(dx * dx + dy * dy + dz * dz); + bool good = dist < radius; + + // Interior holes (positions relative to geometry size so they work at any dim) + const float32 h1cx = cx + radius * 0.3f; + const float32 h1cy = cy + radius * 0.3f; + const float32 h1cz = cz + radius * 0.3f; + const float32 h1r = dimX * 0.053f; // ~4 at 75, ~10.6 at 200 + const float32 h2cx = cx - radius * 0.3f; + const float32 h2cy = cy - radius * 0.3f; + const float32 h2cz = cz - radius * 0.3f; + const float32 h2r = dimX * 0.04f; // ~3 at 75, ~8 at 200 + + if(good) + { + const float32 h1 = std::sqrt((static_cast(x) - h1cx) * (static_cast(x) - h1cx) + (static_cast(y) - h1cy) * (static_cast(y) - h1cy) + + (static_cast(z) - h1cz) * (static_cast(z) - h1cz)); + if(h1 < h1r) + { + good = false; + } + const float32 h2 = std::sqrt((static_cast(x) - h2cx) * (static_cast(x) - h2cx) + (static_cast(y) - h2cy) * (static_cast(y) - h2cy) + + (static_cast(z) - h2cz) * (static_cast(z) - h2cz)); + if(h2 < h2r) + { + good = false; + } + } + + // Isolated noise outside the sphere + if(!good && dist < radius + 5.0f && dist > radius) + { + if((x + y + z) % 7 == 0) + { + good = true; + } + } + + sliceBuffer[y * dimX + x] = good ? 1 : 0; + } + } + maskStore.copyFromBuffer(z * sliceSize, nonstd::span(sliceBuffer.data(), sliceSize)); + } } -TEST_CASE("SimplnxCore::IdentifySampleFilter", "[SimplnxCore][IdentifySampleFilter]") + +/** + * @brief Populates IdentifySampleFilter arguments from a test variant name. + * + * Name convention: "whole_fill", "sliced_xy_nofill", etc. + */ +void SetupArgs(Arguments& args, const std::string& testName, const DataPath& geomPath, const DataPath& maskPath) +{ + const bool fillHoles = (testName.find("nofill") == std::string::npos); + const bool sliceBySlice = (testName.find("sliced") != std::string::npos); + ChoicesParameter::ValueType slicePlane = 0; + if(testName.find("xz") != std::string::npos) + { + slicePlane = 1; + } + else if(testName.find("yz") != std::string::npos) + { + slicePlane = 2; + } + + args.insertOrAssign(IdentifySampleFilter::k_SelectedImageGeometryPath_Key, std::make_any(geomPath)); + args.insertOrAssign(IdentifySampleFilter::k_MaskArrayPath_Key, std::make_any(maskPath)); + args.insertOrAssign(IdentifySampleFilter::k_FillHoles_Key, std::make_any(fillHoles)); + args.insertOrAssign(IdentifySampleFilter::k_SliceBySlice_Key, std::make_any(sliceBySlice)); + args.insertOrAssign(IdentifySampleFilter::k_SliceBySlicePlane_Key, std::make_any(slicePlane)); +} +} // namespace + +TEST_CASE("SimplnxCore::IdentifySampleFilter: 200x200x200 Exemplar Comparison", "[SimplnxCore][IdentifySampleFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // uint8 1-comp => 200*200*1 = 40,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 40000, true); + + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_ExemplarFile); - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "identify_sample_v2.tar.gz", "identify_sample_v2"); - using TestArgType = std::tuple; - /* clang-format off */ - std::vector allTestParams = { - {"sliced", "xy", "fill"}, - {"sliced", "xy", "nofill"}, - {"sliced", "xz", "fill"}, - {"sliced", "xz", "nofill"}, - {"sliced", "yz", "fill"}, - {"sliced", "yz", "nofill"}, - - {"whole", "xy", "fill"}, - {"whole", "xy", "nofill"}, - {"whole", "xz", "fill"}, - {"whole", "xz", "nofill"}, - {"whole", "yz", "fill"}, - {"whole", "yz", "nofill"}, - }; - /* clang-format on */ - for(const auto& testParam : allTestParams) + std::string testName = GENERATE("whole_fill", "whole_nofill", "sliced_xy_fill", "sliced_xy_nofill", "sliced_xz_fill", "sliced_xz_nofill", "sliced_yz_fill", "sliced_yz_nofill"); + DYNAMIC_SECTION("Variant: " << testName) { - std::string slice_by_slice = std::get<0>(testParam); - bool sliceBySlice = slice_by_slice == "sliced"; + DataStructure dataStructure; + BuildIdentifySampleTestData(dataStructure, k_Dim, k_Dim, k_Dim); - std::string slice_plane = std::get<1>(testParam); + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(k_MaskPath)); - ChoicesParameter::ValueType sliceBySlicePlane = 0; - if(slice_plane == "xz") - sliceBySlicePlane = 1; - else if(slice_plane == "yz") - sliceBySlicePlane = 2; + IdentifySampleFilter filter; + Arguments args; + SetupArgs(args, testName, k_GeomPath, k_MaskPath); - std::string fill_holes = std::get<2>(testParam); - bool fillHoles = fill_holes == "fill"; + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - SECTION(fmt::format("{}_{}_{}", slice_by_slice, slice_plane, fill_holes)) - { - fs::path inputFilePath = fs::path(fmt::format("{}/identify_sample_v2/{}_{}_{}.dream3d", unit_test::k_TestFilesDir, slice_by_slice, slice_plane, fill_holes)); - std::cout << inputFilePath.string() << std::endl; - - DataStructure dataStructure = LoadDataStructure(inputFilePath); - IdentifySampleFilter filter; - Arguments args; - args.insert(IdentifySampleFilter::k_SelectedImageGeometryPath_Key, std::make_any(Constants::k_DataContainerPath)); - args.insert(IdentifySampleFilter::k_MaskArrayPath_Key, std::make_any(Constants::k_MaskArrayPath)); - args.insert(IdentifySampleFilter::k_FillHoles_Key, std::make_any(fillHoles)); - args.insert(IdentifySampleFilter::k_SliceBySlice_Key, std::make_any(sliceBySlice)); - args.insert(IdentifySampleFilter::k_SliceBySlicePlane_Key, std::make_any(sliceBySlicePlane)); - - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) - - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) - -#ifdef SIMPLNX_WRITE_TEST_OUTPUT - WriteTestDataStructure(dataStructure, fmt::format("{}/identify_sample_output_{}_{}_{}.dream3d", unit_test::k_BinaryTestOutputDir, fillHoles, sliceBySlice, sliceBySlicePlane)); -#endif - - const IDataArray& computedArray = dataStructure.getDataRefAs(Constants::k_MaskArrayPath); - const IDataArray& exemplarArray = dataStructure.getDataRefAs(k_ExemplarArrayPath); - CompareDataArrays(computedArray, exemplarArray); - - UnitTest::CheckArraysInheritTupleDims(dataStructure); - } + // Compare against exemplar + const std::string exemplarGeomName = testName + "_Exemplar"; + const DataPath exemplarMaskPath({exemplarGeomName, std::string(k_CellDataName), "Mask"}); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_MaskPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarMaskPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarMaskPath), dataStructure.getDataRefAs(k_MaskPath)); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} + +TEST_CASE("SimplnxCore::IdentifySampleFilter: Generate Test Data", "[SimplnxCore][IdentifySampleFilter][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); + + const auto outputDir = fs::path(fmt::format("{}/generated_test_data/identify_sample", unit_test::k_BinaryTestOutputDir)); + fs::create_directories(outputDir); + + DataStructure ds; + for(const auto& name : {"whole_fill", "whole_nofill", "sliced_xy_fill", "sliced_xy_nofill", "sliced_xz_fill", "sliced_xz_nofill", "sliced_yz_fill", "sliced_yz_nofill"}) + { + BuildIdentifySampleTestData(ds, k_Dim, k_Dim, k_Dim, name); } + UnitTest::WriteTestDataStructure(ds, outputDir / "input.dream3d"); } diff --git a/src/Plugins/SimplnxCore/test/QuickSurfaceMeshFilterTest.cpp b/src/Plugins/SimplnxCore/test/QuickSurfaceMeshFilterTest.cpp index 43f901cd73..394b1df972 100644 --- a/src/Plugins/SimplnxCore/test/QuickSurfaceMeshFilterTest.cpp +++ b/src/Plugins/SimplnxCore/test/QuickSurfaceMeshFilterTest.cpp @@ -8,6 +8,7 @@ #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/MultiArraySelectionParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include @@ -19,6 +20,9 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter", "[SimplnxCore][QuickSurfaceMesh { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "QuickSurfaceMeshTest_v2.tar.gz", "QuickSurfaceMeshTest_v2"); // Read the Small IN100 Data set @@ -45,6 +49,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter", "[SimplnxCore][QuickSurfaceMesh Arguments args; QuickSurfaceMeshFilter filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdCellDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(ebsdCellDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -52,6 +57,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter", "[SimplnxCore][QuickSurfaceMesh selectedCellArrayPaths.push_back(ebsdCellDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdFeatureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(ebsdFeatureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -89,6 +95,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter", "[SimplnxCore][QuickSurfaceMesh #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -113,6 +120,9 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding", "[SimplnxCore][QuickSu { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "QuickSurfaceMeshTest_v2.tar.gz", "QuickSurfaceMeshTest_v2"); // Read the Small IN100 Data set @@ -139,6 +149,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding", "[SimplnxCore][QuickSu Arguments args; QuickSurfaceMeshFilter filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdCellDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(ebsdCellDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -146,6 +157,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding", "[SimplnxCore][QuickSu selectedCellArrayPaths.push_back(ebsdCellDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdFeatureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(ebsdFeatureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -183,6 +195,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding", "[SimplnxCore][QuickSu #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -207,6 +220,9 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Problem Voxels", "[SimplnxCore][ { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "QuickSurfaceMeshTest_v2.tar.gz", "QuickSurfaceMeshTest_v2"); // Read the Small IN100 Data set @@ -233,6 +249,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Problem Voxels", "[SimplnxCore][ Arguments args; QuickSurfaceMeshFilter filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdCellDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(ebsdCellDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -240,6 +257,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Problem Voxels", "[SimplnxCore][ selectedCellArrayPaths.push_back(ebsdCellDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdFeatureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(ebsdFeatureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -277,6 +295,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Problem Voxels", "[SimplnxCore][ #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -301,6 +320,9 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding and Problem Voxels", "[S { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "QuickSurfaceMeshTest_v2.tar.gz", "QuickSurfaceMeshTest_v2"); // Read the Small IN100 Data set @@ -327,6 +349,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding and Problem Voxels", "[S Arguments args; QuickSurfaceMeshFilter filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdCellDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(ebsdCellDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -334,6 +357,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding and Problem Voxels", "[S selectedCellArrayPaths.push_back(ebsdCellDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(ebsdFeatureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(ebsdFeatureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -371,6 +395,7 @@ TEST_CASE("SimplnxCore::QuickSurfaceMeshFilter: Winding and Problem Voxels", "[S #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); diff --git a/src/Plugins/SimplnxCore/test/ReadHDF5DatasetTest.cpp b/src/Plugins/SimplnxCore/test/ReadHDF5DatasetTest.cpp index dcddc70542..82cb39859c 100644 --- a/src/Plugins/SimplnxCore/test/ReadHDF5DatasetTest.cpp +++ b/src/Plugins/SimplnxCore/test/ReadHDF5DatasetTest.cpp @@ -94,47 +94,47 @@ void writeHDF5File() // Create the Pointer group auto ptrGroupWriter = fileWriter.createGroup("Pointer"); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); - writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); + writePointer1DArrayDataset(ptrGroupWriter); writePointer1DArrayDataset(ptrGroupWriter); writePointer1DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); - writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); + writePointer2DArrayDataset(ptrGroupWriter); writePointer2DArrayDataset(ptrGroupWriter); writePointer2DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); - writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); + writePointer3DArrayDataset(ptrGroupWriter); writePointer3DArrayDataset(ptrGroupWriter); writePointer3DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); - writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); + writePointer4DArrayDataset(ptrGroupWriter); writePointer4DArrayDataset(ptrGroupWriter); writePointer4DArrayDataset(ptrGroupWriter); } @@ -203,7 +203,7 @@ void testFilterPreflight(ReadHDF5DatasetFilter& filter) // Fill in Dataset Path with a valid path so that we can continue our error checks importInfoList.clear(); - std::string typeStr = nx::core::HDF5::Support::HdfTypeForPrimitiveAsStr(); + std::string typeStr = nx::core::HDF5::Support::HdfTypeForPrimitiveAsStr(); importInfo.dataSetPath = "Pointer/Pointer1DArrayDataset<" + typeStr + ">"; importInfoList.push_back(importInfo); val = {levelZeroPath, m_FilePath, importInfoList}; @@ -260,7 +260,7 @@ void testFilterPreflight(ReadHDF5DatasetFilter& filter) } // ----------------------------------------------------------------------------- -std::string createVectorString(std::vector vec) +std::string createVectorString(const ShapeType& vec) { std::string str = "("; for(int i = 0; i < vec.size(); i++) @@ -340,16 +340,16 @@ void DatasetTest(ReadHDF5DatasetFilter& filter, const std::list tDims = StringUtilities::split(tDimsStr, ','); - size_t tDimsProduct = 1; + usize tDimsProduct = 1; for(const auto& tDim : tDims) { - size_t tdim = std::stoi(tDim); + usize tdim = std::stoi(tDim); tDimsProduct = tDimsProduct * tdim; } std::string cDimsStr = info.componentDimensions; std::vector tokens = StringUtilities::split(cDimsStr, ','); - std::vector cDims; + ShapeType cDims; cDims.reserve(tokens.size()); for(const auto& token : tokens) { @@ -357,8 +357,8 @@ void DatasetTest(ReadHDF5DatasetFilter& filter, const std::listgetNumberOfTuples(); auto daNumComponents = da->getNumberOfComponents(); - size_t totalArrayValues = daNumTuples * daNumComponents; + usize totalArrayValues = daNumTuples * daNumComponents; REQUIRE(totalArrayValues == tDimsProduct * cDimsProduct); - for(size_t i = 0; i < tDimsProduct * cDimsProduct; ++i) + // Bulk-read into local buffer to avoid per-element OOC overhead + std::vector buf(totalArrayValues); + da->getDataStoreRef().copyIntoBuffer(0, nonstd::span(buf.data(), totalArrayValues)); + for(usize i = 0; i < totalArrayValues; ++i) { - T value = da->at(i); + T value = buf[i]; REQUIRE(value == static_cast(i * 5)); } } } + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } // ----------------------------------------------------------------------------- @@ -388,8 +393,8 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) // // ******************* Test Reading Data ************************************* // Create tuple and component dimensions for all tests - std::vector> tDimsVector; - std::vector> cDimsVector; + std::vector tDimsVector; + std::vector cDimsVector; // Add 1D, 2D, 3D, and 4D tuple and component dimensions that test all 4 possibilities: // 1. Tuple dimensions and component dimensions are both valid @@ -397,43 +402,43 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) // 3. Tuple dimensions are invalid, but component dimensions are valid // 4. Neither tuple dimensions or component dimensions are valid - tDimsVector.push_back(std::vector(1) = {TUPLEDIMPROD}); - cDimsVector.push_back(std::vector(1) = {COMPDIMPROD}); + tDimsVector.push_back(ShapeType{TUPLEDIMPROD}); + cDimsVector.push_back(ShapeType{COMPDIMPROD}); - tDimsVector.push_back(std::vector(2) = {10, 4}); - cDimsVector.push_back(std::vector(2) = {12, 6}); + tDimsVector.push_back(ShapeType{10, 4}); + cDimsVector.push_back(ShapeType{12, 6}); - tDimsVector.push_back(std::vector(3) = {2, 2, 10}); - cDimsVector.push_back(std::vector(3) = {4, 3, 6}); + tDimsVector.push_back(ShapeType{2, 2, 10}); + cDimsVector.push_back(ShapeType{4, 3, 6}); - tDimsVector.push_back(std::vector(4) = {2, 2, 5, 2}); - cDimsVector.push_back(std::vector(4) = {4, 3, 3, 2}); + tDimsVector.push_back(ShapeType{2, 2, 5, 2}); + cDimsVector.push_back(ShapeType{4, 3, 3, 2}); - tDimsVector.push_back(std::vector(1) = {TUPLEDIMPROD - 1}); - cDimsVector.push_back(std::vector(1) = {COMPDIMPROD - 1}); + tDimsVector.push_back(ShapeType{TUPLEDIMPROD - 1}); + cDimsVector.push_back(ShapeType{COMPDIMPROD - 1}); - tDimsVector.push_back(std::vector(2) = {TUPLEDIMPROD - 1, 34}); - cDimsVector.push_back(std::vector(2) = {COMPDIMPROD - 1, 56}); + tDimsVector.push_back(ShapeType{TUPLEDIMPROD - 1, 34}); + cDimsVector.push_back(ShapeType{COMPDIMPROD - 1, 56}); - tDimsVector.push_back(std::vector(3) = {TUPLEDIMPROD - 1, 23, 654}); - cDimsVector.push_back(std::vector(3) = {COMPDIMPROD - 1, 56, 12}); + tDimsVector.push_back(ShapeType{TUPLEDIMPROD - 1, 23, 654}); + cDimsVector.push_back(ShapeType{COMPDIMPROD - 1, 56, 12}); - tDimsVector.push_back(std::vector(4) = {TUPLEDIMPROD - 1, 98, 12, 45}); - cDimsVector.push_back(std::vector(4) = {COMPDIMPROD - 1, 43, 12, 53}); + tDimsVector.push_back(ShapeType{TUPLEDIMPROD - 1, 98, 12, 45}); + cDimsVector.push_back(ShapeType{COMPDIMPROD - 1, 43, 12, 53}); // Execute all combinations of tests for(const auto& tDims : tDimsVector) { for(const auto& cDims : cDimsVector) { - size_t amTupleCount = 1; - for(size_t tDim : tDims) + usize amTupleCount = 1; + for(usize tDim : tDims) { amTupleCount *= tDim; } - size_t cDimsProd = 1; - for(size_t cDim : cDims) + usize cDimsProd = 1; + for(usize cDim : cDims) { cDimsProd *= cDim; } @@ -459,14 +464,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) // Run 1D Array Tests info.dataSetPath = dsetPaths[0]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); @@ -475,14 +480,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) // Run 2D Array Tests info.dataSetPath = dsetPaths[1]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); @@ -491,14 +496,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) // Run 3D Array Tests info.dataSetPath = dsetPaths[2]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); @@ -507,14 +512,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) // Run 4D Array Tests info.dataSetPath = dsetPaths[3]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); @@ -530,14 +535,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) info.dataSetPath = dsetPaths[b]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); @@ -559,14 +564,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) info.dataSetPath = dsetPaths[c]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); @@ -587,14 +592,14 @@ void testFilterExecute(ReadHDF5DatasetFilter& filter) info.dataSetPath = dsetPaths[3]; importInfoList.push_back(info); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); - DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); + DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); DatasetTest(filter, importInfoList, true, resultsValid); } diff --git a/src/Plugins/SimplnxCore/test/ReadRawBinaryTest.cpp b/src/Plugins/SimplnxCore/test/ReadRawBinaryTest.cpp index aefcb85b70..6ed135b318 100644 --- a/src/Plugins/SimplnxCore/test/ReadRawBinaryTest.cpp +++ b/src/Plugins/SimplnxCore/test/ReadRawBinaryTest.cpp @@ -115,15 +115,25 @@ void TestCase1_Execute(NumericType scalarType) auto executeResult = filter.execute(dataStructure, args); SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + REQUIRE_NOTHROW(dataStructure.getDataRefAs>(k_CreatedArrayPath)); const DataArray& createdData = dataStructure.getDataRefAs>(k_CreatedArrayPath); const AbstractDataStore& store = createdData.getDataStoreRef(); bool isSame = true; - for(usize i = 0; i < dataArraySize; ++i) { - if(store[i] != exemplaryData[i]) + constexpr usize k_BufSize = 1000000; + std::vector readBuf(std::min(dataArraySize, k_BufSize)); + for(usize start = 0; start < dataArraySize && isSame; start += k_BufSize) { - isSame = false; - break; + usize count = std::min(k_BufSize, dataArraySize - start); + store.copyIntoBuffer(start, nonstd::span(readBuf.data(), count)); + for(usize i = 0; i < count; ++i) + { + if(readBuf[i] != exemplaryData[start + i]) + { + isSame = false; + break; + } + } } } REQUIRE(isSame); @@ -261,12 +271,21 @@ void TestCase4_Execute(NumericType scalarType) constexpr usize elementOffset = skipHeaderBytes / sizeof(T); bool isSame = true; usize size = createdStore.getSize(); - for(usize i = 0; i < size; ++i) { - if(createdStore[i] != exemplaryData[i + elementOffset]) + constexpr usize k_BufSize = 1000000; + std::vector readBuf(std::min(size, k_BufSize)); + for(usize start = 0; start < size && isSame; start += k_BufSize) { - isSame = false; - break; + usize count = std::min(k_BufSize, size - start); + createdStore.copyIntoBuffer(start, nonstd::span(readBuf.data(), count)); + for(usize i = 0; i < count; ++i) + { + if(readBuf[i] != exemplaryData[start + i + elementOffset]) + { + isSame = false; + break; + } + } } } REQUIRE(isSame); diff --git a/src/Plugins/SimplnxCore/test/ReplaceElementAttributesWithNeighborValuesTest.cpp b/src/Plugins/SimplnxCore/test/ReplaceElementAttributesWithNeighborValuesTest.cpp index 5d93378d0d..e4fb6bfa52 100644 --- a/src/Plugins/SimplnxCore/test/ReplaceElementAttributesWithNeighborValuesTest.cpp +++ b/src/Plugins/SimplnxCore/test/ReplaceElementAttributesWithNeighborValuesTest.cpp @@ -3,63 +3,240 @@ #include "SimplnxCore/Filters/ReplaceElementAttributesWithNeighborValuesFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include -namespace fs = std::filesystem; +namespace fs = std::filesystem; using namespace nx::core; using namespace nx::core::Constants; using namespace nx::core::UnitTest; namespace { -const DataPath k_ConfidenceIndexPath = k_CellAttributeMatrix.createChildPath(Constants::k_Confidence_Index); -const std::string k_ExemplarDataContainer2("DataContainer"); +const std::string k_GeomName("DataContainer"); +const std::string k_CellDataName("CellData"); + +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_CellDataPath = k_GeomPath.createChildPath(k_CellDataName); +const DataPath k_ConfidencePath = k_CellDataPath.createChildPath("Confidence Index"); + +void BuildTestData(DataStructure& dataStructure, usize dimX, usize dimY, usize dimZ) +{ + const ShapeType cellTupleShape = {dimZ, dimY, dimX}; + const usize sliceSize = dimX * dimY; + + auto* imageGeom = ImageGeom::Create(dataStructure, k_GeomName); + imageGeom->setDimensions({dimX, dimY, dimZ}); + imageGeom->setSpacing({1.0f, 1.0f, 1.0f}); + imageGeom->setOrigin({0.0f, 0.0f, 0.0f}); + + auto* cellAM = AttributeMatrix::Create(dataStructure, k_CellDataName, cellTupleShape, imageGeom->getId()); + imageGeom->setCellData(*cellAM); + + auto confDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* confArray = DataArray::Create(dataStructure, "Confidence Index", confDataStore, cellAM->getId()); + auto& confStore = confArray->getDataStoreRef(); + + auto eulerDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {3}, IDataAction::Mode::Execute); + auto* eulerArray = DataArray::Create(dataStructure, "EulerAngles", eulerDataStore, cellAM->getId()); + auto& eulerStore = eulerArray->getDataStoreRef(); + + auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto* phasesArray = DataArray::Create(dataStructure, "Phases", phasesDataStore, cellAM->getId()); + auto& phasesStore = phasesArray->getDataStoreRef(); + + std::vector confBuf(sliceSize); + std::vector eulerBuf(sliceSize * 3); + std::vector phasesBuf(sliceSize); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize inSlice = y * dimX + x; + phasesBuf[inSlice] = 1; + + confBuf[inSlice] = static_cast((x * 3 + y * 7 + z * 11) % 100) / 100.0f; + + const usize eIdx = inSlice * 3; + eulerBuf[eIdx] = static_cast(x) / static_cast(dimX); + eulerBuf[eIdx + 1] = static_cast(y) / static_cast(dimY); + eulerBuf[eIdx + 2] = static_cast(z) / static_cast(dimZ); + } + } + const usize zOffset = z * sliceSize; + confStore.copyFromBuffer(zOffset, nonstd::span(confBuf.data(), sliceSize)); + eulerStore.copyFromBuffer(zOffset * 3, nonstd::span(eulerBuf.data(), sliceSize * 3)); + phasesStore.copyFromBuffer(zOffset, nonstd::span(phasesBuf.data(), sliceSize)); + } +} + +usize CountVoxelsBelowThreshold(const DataStructure& dataStructure, float32 threshold, usize dimX, usize dimY, usize dimZ) +{ + const auto& conf = dataStructure.getDataRefAs(k_ConfidencePath).getDataStoreRef(); + const usize sliceSize = dimX * dimY; + std::vector buf(sliceSize); + usize count = 0; + for(usize z = 0; z < dimZ; z++) + { + conf.copyIntoBuffer(z * sliceSize, nonstd::span(buf.data(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(buf[i] < threshold) + { + count++; + } + } + } + return count; +} + +usize CountVoxelsAboveThreshold(const DataStructure& dataStructure, float32 threshold, usize dimX, usize dimY, usize dimZ) +{ + const auto& conf = dataStructure.getDataRefAs(k_ConfidencePath).getDataStoreRef(); + const usize sliceSize = dimX * dimY; + std::vector buf(sliceSize); + usize count = 0; + for(usize z = 0; z < dimZ; z++) + { + conf.copyIntoBuffer(z * sliceSize, nonstd::span(buf.data(), sliceSize)); + for(usize i = 0; i < sliceSize; i++) + { + if(buf[i] > threshold) + { + count++; + } + } + } + return count; +} } // namespace -TEST_CASE("SimplnxCore::ReplaceElementAttributesWithNeighborValuesFilter", "[SimplnxCore][ReplaceElementAttributesWithNeighborValuesFilter]") +TEST_CASE("SimplnxCore::ReplaceElementAttributesWithNeighborValuesFilter: Small Correctness", "[SimplnxCore][ReplaceElementAttributesWithNeighborValuesFilter]") { UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 20x20x20, EulerAngles (float32, 3-comp) => 20*20*3*4 = 4,800 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 4800, true); + + auto comparison = GENERATE(0ULL, 1ULL); + bool loopUntilDone = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + + std::string compName = (comparison == 0) ? "LessThan" : "GreaterThan"; + std::string loopName = loopUntilDone ? "Loop" : "NoLoop"; + DYNAMIC_SECTION(compName << " " << loopName << " forceOoc: " << forceOocAlgo) + { + constexpr float32 k_Threshold = 0.1F; + + DataStructure dataStructure; + BuildTestData(dataStructure, 20, 20, 20); + + const usize belowCountBefore = CountVoxelsBelowThreshold(dataStructure, k_Threshold, 20, 20, 20); + const usize aboveCountBefore = CountVoxelsAboveThreshold(dataStructure, k_Threshold, 20, 20, 20); + REQUIRE(belowCountBefore > 0); + REQUIRE(aboveCountBefore > 0); + + { + ReplaceElementAttributesWithNeighborValuesFilter filter; + Arguments args; + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_MinConfidence_Key, std::make_any(k_Threshold)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_SelectedComparison_Key, std::make_any(comparison)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_Loop_Key, std::make_any(loopUntilDone)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_ComparisonDataPath, std::make_any(k_ConfidencePath)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + } + + const usize belowCountAfter = CountVoxelsBelowThreshold(dataStructure, k_Threshold, 20, 20, 20); + const usize aboveCountAfter = CountVoxelsAboveThreshold(dataStructure, k_Threshold, 20, 20, 20); + + if(comparison == 0) + { + REQUIRE(belowCountAfter <= belowCountBefore); + } + else + { + REQUIRE(aboveCountAfter <= aboveCountBefore); + } + + // TODO: Add exemplar comparison after exemplar archive is published + + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } +} - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_6_replace_element_attributes_with_neighbor.tar.gz", - "6_6_replace_element_attributes_with_neighbor"); +TEST_CASE("SimplnxCore::ReplaceElementAttributesWithNeighborValuesFilter: Generate Test Data", "[SimplnxCore][ReplaceElementAttributesWithNeighborValuesFilter][.GenerateTestData]") +{ + const auto outputDir = fs::path(unit_test::k_BinaryTestOutputDir.view()) / "generated_test_data" / "replace_element_attributes"; + fs::create_directories(outputDir); - // Read Exemplar DREAM3D File Filter - auto exemplarFilePath = fs::path(fmt::format("{}/TestFiles/6_6_replace_element_attributes_with_neighbor/6_6_replace_element_attributes_with_neighbor.dream3d", unit_test::k_DREAM3DDataDir)); - DataStructure exemplarDataStructure = nx::core::UnitTest::LoadDataStructure(exemplarFilePath); + // Small input data (20x20x20) + { + DataStructure buildDS; + BuildTestData(buildDS, 20, 20, 20); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "small_input.dream3d"); + fmt::print("Generated small input: {}\n", (outputDir / "small_input.dream3d").string()); + } + + // Large input data (200x200x200) + { + DataStructure buildDS; + BuildTestData(buildDS, 200, 200, 200); + UnitTest::WriteTestDataStructure(buildDS, outputDir / "large_input.dream3d"); + fmt::print("Generated large input: {}\n", (outputDir / "large_input.dream3d").string()); + } +} - // Read the Test Data set - auto baseDataFilePath = fs::path(fmt::format("{}/TestFiles/6_6_replace_element_attributes_with_neighbor/6_6_replace_element_attributes_with_neighbor.dream3d", unit_test::k_DREAM3DDataDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); +TEST_CASE("SimplnxCore::ReplaceElementAttributesWithNeighborValuesFilter: 200x200x200 Large OOC", "[SimplnxCore][ReplaceElementAttributesWithNeighborValuesFilter]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // 200x200x200, EulerAngles (float32, 3-comp) => 200*200*3*4 = 480,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 480000, true); + DYNAMIC_SECTION("forceOoc: " << forceOocAlgo) { - // Instantiate the filter, a DataStructure object and an Arguments Object + constexpr float32 k_Threshold = 0.1F; + + DataStructure dataStructure; + BuildTestData(dataStructure, 200, 200, 200); + + const usize belowCountBefore = CountVoxelsBelowThreshold(dataStructure, k_Threshold, 200, 200, 200); + REQUIRE(belowCountBefore > 0); + ReplaceElementAttributesWithNeighborValuesFilter filter; Arguments args; - - // Create default Parameters for the filter. - args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_MinConfidence_Key, std::make_any(0.1F)); - args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_SelectedComparison_Key, std::make_any(0)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_MinConfidence_Key, std::make_any(k_Threshold)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_SelectedComparison_Key, std::make_any(0ULL)); args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_Loop_Key, std::make_any(true)); - args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_ComparisonDataPath, std::make_any(k_ConfidenceIndexPath)); - args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_DataContainerPath)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_ComparisonDataPath, std::make_any(k_ConfidencePath)); + args.insertOrAssign(ReplaceElementAttributesWithNeighborValuesFilter::k_SelectedImageGeometryPath_Key, std::make_any(k_GeomPath)); - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); - // Execute the filter and check the result auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) - } + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - UnitTest::CompareExemplarToGeneratedData(dataStructure, exemplarDataStructure, k_CellAttributeMatrix, k_ExemplarDataContainer2); + const usize belowCountAfter = CountVoxelsBelowThreshold(dataStructure, k_Threshold, 200, 200, 200); + REQUIRE(belowCountAfter <= belowCountBefore); -#ifdef SIMPLNX_WRITE_TEST_OUTPUT - WriteTestDataStructure(dataStructure, fmt::format("{}/7_0_replace_element_attributes_with_neighbor.dream3d", unit_test::k_BinaryTestOutputDir)); -#endif - - UnitTest::CheckArraysInheritTupleDims(dataStructure); + UnitTest::CheckArraysInheritTupleDims(dataStructure); + } } diff --git a/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp b/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp index 00cb47718f..4ef3a6fb0d 100644 --- a/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp +++ b/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp @@ -1,173 +1,306 @@ #include "SimplnxCore/Filters/ScalarSegmentFeaturesFilter.hpp" #include "SimplnxCore/SimplnxCore_test_dirs.hpp" -#include "simplnx/DataStructure/IO/HDF5/DataStructureWriter.hpp" -#include "simplnx/Parameters/ArrayCreationParameter.hpp" -#include "simplnx/Parameters/BoolParameter.hpp" +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" #include "simplnx/Parameters/ChoicesParameter.hpp" +#include "simplnx/UnitTest/SegmentFeaturesTestUtils.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" -#include "simplnx/Utilities/DataArrayUtilities.hpp" -#include "simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.hpp" -#include "simplnx/Utilities/Parsing/HDF5/IO/FileIO.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include +#include + using namespace nx::core; using namespace nx::core::UnitTest; -using namespace nx::core::Constants; namespace { -const std::string k_SharedEdgesInputArrayName = "Shared Edges"; -const std::string k_SharedPointsInputArrayName = "Shared Points"; -const std::string k_NothingSharedInputArrayName = "Nothing Shared"; -const std::string k_CombinationInputArrayName = "Combination"; -const std::string k_ExemplarySharedEdgesFaceOnlyFeatureIdsName = "Exemplary Shared Edges FeatureIds - Face Only"; -const std::string k_ExemplarySharedEdgesAllConnectedFeatureIdsName = "Exemplary Shared Edges FeatureIds - All Connected"; -const std::string k_ExemplarySharedPointsFaceOnlyFeatureIdsName = "Exemplary Shared Points FeatureIds - Face Only"; -const std::string k_ExemplarySharedPointsAllConnectedFeatureIdsName = "Exemplary Shared Points FeatureIds - All Connected"; -const std::string k_ExemplaryNothingSharedFaceOnlyFeatureIdsName = "Exemplary Nothing Shared FeatureIds - Face Only"; -const std::string k_ExemplaryNothingSharedAllConnectedFeatureIdsName = "Exemplary Nothing Shared FeatureIds - All Connected"; -const std::string k_ExemplaryCombinationFaceOnlyFeatureIdsName = "Exemplary Combination FeatureIds - Face Only"; -const std::string k_ExemplaryCombinationAllConnectedFeatureIdsName = "Exemplary Combination FeatureIds - All Connected"; +// Exemplar archive +const std::string k_ArchiveName = "segment_features_exemplars.tar.gz"; +const std::string k_DataDirName = "segment_features_exemplars"; +const fs::path k_DataDir = fs::path(unit_test::k_TestFilesDir.view()) / k_DataDirName; +const fs::path k_SmallExemplarFile = k_DataDir / "scalar_small.dream3d"; +const fs::path k_LargeExemplarFile = k_DataDir / "scalar_large.dream3d"; + +// Geometry names +constexpr StringLiteral k_GeomName = "DataContainer"; +constexpr StringLiteral k_CellDataName = "CellData"; +constexpr StringLiteral k_FeatureDataName = "CellFeatureData"; + +// Output array paths +const DataPath k_GeomPath({k_GeomName}); +const DataPath k_FeatureIdsPath({k_GeomName, k_CellDataName, "FeatureIds"}); +const DataPath k_ActivePath({k_GeomName, k_FeatureDataName, "Active"}); +const DataPath k_MaskPath({k_GeomName, k_CellDataName, "Mask"}); + +// Test dimensions +constexpr usize k_SmallDim = 15; +constexpr usize k_SmallBlockSize = 5; +constexpr usize k_LargeDim = 200; +constexpr usize k_LargeBlockSize = 25; + +/** + * @brief Populates ScalarSegmentFeaturesFilter arguments. + */ +void SetupArgs(Arguments& args, bool useMask, bool isPeriodic, int tolerance, ChoicesParameter::ValueType neighborScheme = 0, bool randomize = false) +{ + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_GridGeomPath_Key, std::make_any(k_GeomPath)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_InputArrayPathKey, std::make_any(DataPath({k_GeomName, k_CellDataName, "ScalarData"}))); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ScalarToleranceKey, std::make_any(tolerance)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_UseMask_Key, std::make_any(useMask)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(useMask ? k_MaskPath : DataPath{})); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborScheme)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_IsPeriodic_Key, std::make_any(isPeriodic)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_FeatureIdsName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_CellFeatureName_Key, std::make_any(std::string(k_FeatureDataName))); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_RandomizeFeatures_Key, std::make_any(randomize)); +} } // namespace -TEST_CASE("SimplnxCore::ScalarSegmentFeatures", "[SimplnxCore][ScalarSegmentFeatures]") +TEST_CASE("SimplnxCore::ScalarSegmentFeatures: Small Correctness", "[SimplnxCore][ScalarSegmentFeatures]") { - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "6_5_test_data_1_v2.tar.gz", "6_5_test_data_1_v2"); + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // int32 1-comp => 15*15*4 = 900 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 900, true); - // Read the Small IN100 Data set - auto baseDataFilePath = fs::path(fmt::format("{}/6_5_test_data_1_v2/6_5_test_data_1_v2.dream3d", nx::core::unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_SmallExemplarFile); + std::string testName = GENERATE("Base", "Masked", "Periodic", "Tolerance"); + DYNAMIC_SECTION("Variant: " << testName) { - Arguments args; + const bool useMask = (testName == "Masked"); + const bool isPeriodic = (testName == "Periodic"); + const int tolerance = (testName == "Tolerance") ? 1 : 0; + + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, {k_SmallDim, k_SmallDim, k_SmallDim}, std::string(k_GeomName), std::string(k_CellDataName)); + BuildScalarTestData(dataStructure, cellShape, am->getId(), k_SmallBlockSize, "ScalarData", isPeriodic); + + if(useMask) + { + BuildSphericalMask(dataStructure, cellShape, am->getId()); + } + + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({k_GeomName, k_CellDataName, "ScalarData"}))); + ScalarSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, useMask, isPeriodic, tolerance); - DataPath smallIn100Group({k_DataContainer}); - DataPath ebsdScanDataPath = smallIn100Group.createChildPath(k_CellData); - DataPath inputDataArrayPath = ebsdScanDataPath.createChildPath(k_FeatureIds); - std::string outputFeatureIdsName = "Output_Feature_Ids"; - std::string computedCellDataName = "Computed_CellData"; - DataPath outputFeatureIdsPath = ebsdScanDataPath.createChildPath(outputFeatureIdsName); - DataPath featureDataGroupPath = smallIn100Group.createChildPath(computedCellDataName); - DataPath activeArrayDataPath = featureDataGroupPath.createChildPath(k_ActiveName); - - DataPath gridGeomDataPath({k_DataContainer}); - int scalarTolerance = 0; - - // Create default Parameters for the filter. - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_GridGeomPath_Key, std::make_any(gridGeomDataPath)); - // Turn off the use of a Mask Array - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(DataPath{})); - // Set the input array and the tolerance - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_InputArrayPathKey, std::make_any(inputDataArrayPath)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ScalarToleranceKey, std::make_any(scalarTolerance)); - // Set the paths to the created arrays - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_FeatureIdsName_Key, std::make_any(outputFeatureIdsName)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_CellFeatureName_Key, std::make_any(computedCellDataName)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - // Are we going to randomize the featureIds when completed. - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_RandomizeFeatures_Key, std::make_any(true)); - - // Preflight the filter and check result auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) - - // Execute the filter and check the result + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); - UInt8Array& actives = dataStructure.getDataRefAs(activeArrayDataPath); - size_t numFeatures = actives.getNumberOfTuples(); - REQUIRE(numFeatures == 847); - } + // Compare against exemplar + const std::string exemplarGeomName = testName + "_Exemplar"; + const DataPath exemplarFeatureIdsPath({exemplarGeomName, std::string(k_CellDataName), "FeatureIds"}); + const DataPath exemplarActivePath({exemplarGeomName, std::string(k_FeatureDataName), "Active"}); - { - // Write out the DataStructure for later viewing/debugging - std::string filePath = fmt::format("{}/ScalarSegmentFeatures.dream3d", unit_test::k_BinaryTestOutputDir); - // std::cout << "Writing file to: " << filePath << std::endl; - nx::core::HDF5::FileIO fileWriter = nx::core::HDF5::FileIO::WriteFile(filePath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(k_FeatureIdsPath)); - auto resultH5 = HDF5::DataStructureWriter::WriteFile(dataStructure, fileWriter); - SIMPLNX_RESULT_REQUIRE_VALID(resultH5); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarActivePath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarActivePath), dataStructure.getDataRefAs(k_ActivePath)); + + UnitTest::CheckArraysInheritTupleDims(dataStructure); } +} + +TEST_CASE("SimplnxCore::ScalarSegmentFeatures: FaceEdgeVertex Connectivity", "[SimplnxCore][ScalarSegmentFeatures]") +{ + UnitTest::LoadPlugins(); + + // Shared test: verifies vertex and edge connectivity with FaceEdgeVertex scheme. + // Setup lambda creates ScalarData with 4 isolated voxels (2 pairs) and configures args. + auto setupScalar = [](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, ChoicesParameter::ValueType neighborScheme) { + const ShapeType cellShape = {3, 3, 3}; + auto& am = ds.getDataRefAs(cellDataPath); + auto scalarDS = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* scalarArr = DataArray::Create(ds, "ScalarData", scalarDS, am.getId()); + auto& store = scalarArr->getDataStoreRef(); + store.fill(0); + store[0 * 9 + 0 * 3 + 0] = 1; // (0,0,0) — vertex pair A + store[1 * 9 + 1 * 3 + 1] = 1; // (1,1,1) — vertex pair B + store[0 * 9 + 0 * 3 + 2] = 2; // (2,0,0) — edge pair C + store[1 * 9 + 1 * 3 + 2] = 2; // (2,1,1) — edge pair D + + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_GridGeomPath_Key, std::make_any(geomPath)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_InputArrayPathKey, std::make_any(cellDataPath.createChildPath("ScalarData"))); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ScalarToleranceKey, std::make_any(0)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(DataPath{})); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborScheme)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_IsPeriodic_Key, std::make_any(false)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_FeatureIdsName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_CellFeatureName_Key, std::make_any("CellFeatureData")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_RandomizeFeatures_Key, std::make_any(false)); + }; + + RunFaceEdgeVertexConnectivityTest([&](Arguments& args, DataStructure& ds, const DataPath& gp, const DataPath& cp) { setupScalar(args, ds, gp, cp, 0); }, + [&](Arguments& args, DataStructure& ds, const DataPath& gp, const DataPath& cp) { setupScalar(args, ds, gp, cp, 1); }); +} + +TEST_CASE("SimplnxCore::ScalarSegmentFeatures: 200x200x200 Large OOC", "[SimplnxCore][ScalarSegmentFeatures]") +{ + UnitTest::LoadPlugins(); + // Test both algorithm paths (in-core + OOC) by default; controlled by CMake SIMPLNX_TEST_ALGORITHM_PATH + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); + // int32 1-comp => 200*200*4 = 160,000 bytes/slice + const UnitTest::PreferencesSentinel prefsSentinel("HDF5-OOC", 160000, true); + + const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, k_ArchiveName, k_DataDirName); + DataStructure exemplarDS = UnitTest::LoadDataStructure(k_LargeExemplarFile); + + const ShapeType cellShape = {k_LargeDim, k_LargeDim, k_LargeDim}; + + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, {k_LargeDim, k_LargeDim, k_LargeDim}, std::string(k_GeomName), std::string(k_CellDataName)); + BuildScalarTestData(dataStructure, cellShape, am->getId(), k_LargeBlockSize, "ScalarData", true); + BuildSphericalMask(dataStructure, cellShape, am->getId()); + + UnitTest::RequireExpectedStoreType(dataStructure.getDataRefAs(DataPath({k_GeomName, k_CellDataName, "ScalarData"}))); + + ScalarSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/true, /*isPeriodic=*/true, /*tolerance=*/0); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + // Compare against exemplar + const DataPath exemplarFeatureIdsPath({"DataContainer_Exemplar", std::string(k_CellDataName), "FeatureIds"}); + const DataPath exemplarActivePath({"DataContainer_Exemplar", std::string(k_FeatureDataName), "Active"}); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarFeatureIdsPath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarFeatureIdsPath), dataStructure.getDataRefAs(k_FeatureIdsPath)); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + REQUIRE_NOTHROW(exemplarDS.getDataRefAs(exemplarActivePath)); + CompareDataArrays(exemplarDS.getDataRefAs(exemplarActivePath), dataStructure.getDataRefAs(k_ActivePath)); UnitTest::CheckArraysInheritTupleDims(dataStructure); } -TEST_CASE("SimplnxCore::ScalarSegmentFeatures: Neighbor Scheme", "[Reconstruction][ScalarSegmentFeatures]") +TEST_CASE("SimplnxCore::ScalarSegmentFeatures: No Valid Voxels Returns Error", "[SimplnxCore][ScalarSegmentFeatures]") +{ + UnitTest::LoadPlugins(); + + RunNoValidVoxelsErrorTest([](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, const DataPath& maskPath) { + const ShapeType cellShape = {3, 3, 3}; + auto& am = ds.getDataRefAs(cellDataPath); + CreateTestDataArray(ds, "ScalarData", cellShape, {1}, am.getId()); + + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_GridGeomPath_Key, std::make_any(geomPath)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_InputArrayPathKey, std::make_any(cellDataPath.createChildPath("ScalarData"))); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ScalarToleranceKey, std::make_any(0)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_UseMask_Key, std::make_any(true)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(maskPath)); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_FeatureIdsName_Key, std::make_any("FeatureIds")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_CellFeatureName_Key, std::make_any("FeatureData")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any("Active")); + args.insertOrAssign(ScalarSegmentFeaturesFilter::k_RandomizeFeatures_Key, std::make_any(false)); + }); +} + +TEST_CASE("SimplnxCore::ScalarSegmentFeatures: Randomize Feature IDs", "[SimplnxCore][ScalarSegmentFeatures]") { - /** - * We are going to use Catch2's GENERATE macro to create variations of parameter values. - * EVERYTHING after the GENERATE macro will be run for each of the generated sets of values - */ - auto [sectionName, inputDataArrayName, exemplaryFeatureIdsArrayName, neighborSchemeIndex] = - GENERATE(std::make_tuple("Shared Edges - Face Only", k_SharedEdgesInputArrayName, k_ExemplarySharedEdgesFaceOnlyFeatureIdsName, 0), - std::make_tuple("Shared Edges - All Connected", k_SharedEdgesInputArrayName, k_ExemplarySharedEdgesAllConnectedFeatureIdsName, 1), - std::make_tuple("Shared Points - Face Only", k_SharedPointsInputArrayName, k_ExemplarySharedPointsFaceOnlyFeatureIdsName, 0), - std::make_tuple("Shared Points - All Connected", k_SharedPointsInputArrayName, k_ExemplarySharedPointsAllConnectedFeatureIdsName, 1), - std::make_tuple("Nothing Shared - Face Only", k_NothingSharedInputArrayName, k_ExemplaryNothingSharedFaceOnlyFeatureIdsName, 0), - std::make_tuple("Nothing Shared - All Connected", k_NothingSharedInputArrayName, k_ExemplaryNothingSharedAllConnectedFeatureIdsName, 1), - std::make_tuple("Combination - Face Only", k_CombinationInputArrayName, k_ExemplaryCombinationFaceOnlyFeatureIdsName, 0), - std::make_tuple("Combination - All Connected", k_CombinationInputArrayName, k_ExemplaryCombinationAllConnectedFeatureIdsName, 1)); - - /** - * @note EVERYTHING from here to the end of the test will be run for **each** tuple set above - */ - const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "segment_features_neighbor_scheme_test.tar.gz", "segment_features_neighbor_scheme_test"); - auto baseDataFilePath = fs::path(fmt::format("{}/segment_features_neighbor_scheme_test/segment_features_neighbor_scheme_test.dream3d", nx::core::unit_test::k_TestFilesDir)); - DataStructure dataStructure = UnitTest::LoadDataStructure(baseDataFilePath); + UnitTest::LoadPlugins(); + + constexpr usize k_ExpectedFeatures = 27; // 3^3 + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + + DataStructure dataStructure; + auto* am = BuildSegmentFeaturesTestGeometry(dataStructure, {k_SmallDim, k_SmallDim, k_SmallDim}, std::string(k_GeomName), std::string(k_CellDataName)); + BuildScalarTestData(dataStructure, cellShape, am->getId(), k_SmallBlockSize); + + ScalarSegmentFeaturesFilter filter; + Arguments args; + SetupArgs(args, /*useMask=*/false, /*isPeriodic=*/false, /*tolerance=*/0, /*neighborScheme=*/0, /*randomize=*/true); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_ActivePath)); + const auto& actives = dataStructure.getDataRefAs(k_ActivePath); + REQUIRE(actives.getNumberOfTuples() == k_ExpectedFeatures + 1); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(k_FeatureIdsPath)); + const auto& featureIds = dataStructure.getDataRefAs(k_FeatureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + std::set uniqueIds; + int32 minId = std::numeric_limits::max(); + int32 maxId = std::numeric_limits::min(); + for(usize i = 0; i < featureStore.getNumberOfTuples(); i++) { - Arguments args; - ScalarSegmentFeaturesFilter filter; + int32 fid = featureStore.getValue(i); + uniqueIds.insert(fid); + minId = std::min(minId, fid); + maxId = std::max(maxId, fid); + } + REQUIRE(minId == 1); + REQUIRE(maxId == static_cast(k_ExpectedFeatures)); + REQUIRE(uniqueIds.size() == k_ExpectedFeatures); +} - DataPath smallIn100Group({k_SmallIn100ImageGeom}); - DataPath ebsdScanDataPath = smallIn100Group.createChildPath(k_Cell_Data); - std::string outputFeatureIdsName = "Output_Feature_Ids"; - std::string computedCellDataName = "Computed_CellData"; - DataPath outputFeatureIdsPath = ebsdScanDataPath.createChildPath(outputFeatureIdsName); - DataPath featureDataGroupPath = smallIn100Group.createChildPath(computedCellDataName); - DataPath activeArrayDataPath = featureDataGroupPath.createChildPath(k_ActiveName); +TEST_CASE("SimplnxCore::ScalarSegmentFeatures: Generate Test Data", "[SimplnxCore][ScalarSegmentFeatures][.GenerateTestData]") +{ + UnitTest::LoadPlugins(); - DataPath gridGeomDataPath({k_SmallIn100ImageGeom}); - int scalarTolerance = 0; + const auto outputDir = fs::path(fmt::format("{}/generated_test_data/scalar_segment_features", unit_test::k_BinaryTestOutputDir)); + fs::create_directories(outputDir); - // Create default Parameters for the filter. - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_GridGeomPath_Key, std::make_any(gridGeomDataPath)); - // Turn off the use of a Mask Array - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_UseMask_Key, std::make_any(false)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_MaskArrayPath_Key, std::make_any(DataPath{})); - // Set the tolerance - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ScalarToleranceKey, std::make_any(scalarTolerance)); - // Set the paths to the created arrays - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_FeatureIdsName_Key, std::make_any(outputFeatureIdsName)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_CellFeatureName_Key, std::make_any(computedCellDataName)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_ActiveArrayName_Key, std::make_any(k_ActiveName)); - // Are we going to randomize the featureIds when completed. - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_RandomizeFeatures_Key, std::make_any(false)); + // Small input data (15^3) — one geometry per test variant + { + const ShapeType cellShape = {k_SmallDim, k_SmallDim, k_SmallDim}; + const std::array dims = {k_SmallDim, k_SmallDim, k_SmallDim}; - SECTION(sectionName) - { - DataPath inputDataArrayPath = ebsdScanDataPath.createChildPath(inputDataArrayName); - DataPath exemplaryFeatureIdsArrayPath = ebsdScanDataPath.createChildPath(exemplaryFeatureIdsArrayName); - DataPath computedFeatureIdsPath = ebsdScanDataPath.createChildPath(outputFeatureIdsName); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_InputArrayPathKey, std::make_any(inputDataArrayPath)); - args.insertOrAssign(ScalarSegmentFeaturesFilter::k_NeighborScheme_Key, std::make_any(neighborSchemeIndex)); + DataStructure ds; - // Preflight the filter and check result - auto preflightResult = filter.preflight(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions) + auto* amBase = BuildSegmentFeaturesTestGeometry(ds, dims, "Base", std::string(k_CellDataName)); + BuildScalarTestData(ds, cellShape, amBase->getId(), k_SmallBlockSize); - // Execute the filter and check the result - auto executeResult = filter.execute(dataStructure, args); - SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result) + auto* amMasked = BuildSegmentFeaturesTestGeometry(ds, dims, "Masked", std::string(k_CellDataName)); + BuildScalarTestData(ds, cellShape, amMasked->getId(), k_SmallBlockSize); + BuildSphericalMask(ds, cellShape, amMasked->getId()); - UnitTest::CompareArrays(dataStructure, exemplaryFeatureIdsArrayPath, computedFeatureIdsPath); + auto* amPeriodic = BuildSegmentFeaturesTestGeometry(ds, dims, "Periodic", std::string(k_CellDataName)); + BuildScalarTestData(ds, cellShape, amPeriodic->getId(), k_SmallBlockSize, "ScalarData", true); - UnitTest::CheckArraysInheritTupleDims(dataStructure); - } + auto* amTolerance = BuildSegmentFeaturesTestGeometry(ds, dims, "Tolerance", std::string(k_CellDataName)); + BuildScalarTestData(ds, cellShape, amTolerance->getId(), k_SmallBlockSize); + + UnitTest::WriteTestDataStructure(ds, outputDir / "small_input.dream3d"); + } + + // Large input data (200^3) — mask=true, periodic=true + { + const ShapeType cellShape = {k_LargeDim, k_LargeDim, k_LargeDim}; + const std::array dims = {k_LargeDim, k_LargeDim, k_LargeDim}; + + DataStructure ds; + auto* am = BuildSegmentFeaturesTestGeometry(ds, dims, std::string(k_GeomName), std::string(k_CellDataName)); + BuildScalarTestData(ds, cellShape, am->getId(), k_LargeBlockSize, "ScalarData", true); + BuildSphericalMask(ds, cellShape, am->getId()); + + UnitTest::WriteTestDataStructure(ds, outputDir / "large_input.dream3d"); } } diff --git a/src/Plugins/SimplnxCore/test/SurfaceNetsTest.cpp b/src/Plugins/SimplnxCore/test/SurfaceNetsTest.cpp index 8b8ce2d9b8..862269ad5b 100644 --- a/src/Plugins/SimplnxCore/test/SurfaceNetsTest.cpp +++ b/src/Plugins/SimplnxCore/test/SurfaceNetsTest.cpp @@ -6,6 +6,7 @@ #include "simplnx/Parameters/BoolParameter.hpp" #include "simplnx/Parameters/MultiArraySelectionParameter.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include @@ -16,6 +17,8 @@ using namespace nx::core::Constants; TEST_CASE("SimplnxCore::SurfaceNetsFilter: Default", "[SimplnxCore][SurfaceNetsFilter]") { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "SurfaceNetsTest_v3.tar.gz", "SurfaceNetsTest_v3"); @@ -43,6 +46,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Default", "[SimplnxCore][SurfaceNetsF Arguments args; SurfaceNetsFilter const filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(celDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(celDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -51,6 +55,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Default", "[SimplnxCore][SurfaceNetsF selectedCellArrayPaths.push_back(celDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(featureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(featureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -92,6 +97,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Default", "[SimplnxCore][SurfaceNetsF #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -100,7 +106,9 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Default", "[SimplnxCore][SurfaceNetsF REQUIRE(vertices->getNumberOfTuples() == 319447); // Compare the shared vertex list and shared triangle list + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarSharedTriPath)); auto& exemplarDataArray = dataStructure.getDataRefAs(exemplarSharedTriPath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName))); auto& computedDataArray = dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName)); CompareDataArrays(exemplarDataArray, computedDataArray); CompareArrays(dataStructure, exemplarSharedVertexPath, computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedVertexListName)); @@ -117,6 +125,8 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Default", "[SimplnxCore][SurfaceNetsF TEST_CASE("SimplnxCore::SurfaceNetsFilter: Smoothing", "[SimplnxCore][SurfaceNetsFilter]") { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "SurfaceNetsTest_v3.tar.gz", "SurfaceNetsTest_v3"); @@ -144,6 +154,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Smoothing", "[SimplnxCore][SurfaceNet Arguments args; SurfaceNetsFilter const filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(celDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(celDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -151,6 +162,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Smoothing", "[SimplnxCore][SurfaceNet selectedCellArrayPaths.push_back(celDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(featureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(featureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -192,6 +204,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Smoothing", "[SimplnxCore][SurfaceNet } // Check a few things about the generated data. { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -200,7 +213,9 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Smoothing", "[SimplnxCore][SurfaceNet } // Compare the shared vertex list and shared triangle list + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarSharedTriPath)); auto& exemplarDataArray = dataStructure.getDataRefAs(exemplarSharedTriPath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName))); auto& computedDataArray = dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName)); CompareDataArrays(exemplarDataArray, computedDataArray); CompareArrays(dataStructure, exemplarSharedVertexPath, computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedVertexListName)); @@ -217,6 +232,8 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Smoothing", "[SimplnxCore][SurfaceNet TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding", "[SimplnxCore][SurfaceNetsFilter]") { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "SurfaceNetsTest_v3.tar.gz", "SurfaceNetsTest_v3"); @@ -244,6 +261,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding", "[SimplnxCore][SurfaceNetsF Arguments args; SurfaceNetsFilter const filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(celDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(celDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -251,6 +269,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding", "[SimplnxCore][SurfaceNetsF selectedCellArrayPaths.push_back(celDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(featureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(featureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -291,6 +310,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding", "[SimplnxCore][SurfaceNetsF #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -299,7 +319,9 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding", "[SimplnxCore][SurfaceNetsF REQUIRE(vertices->getNumberOfTuples() == 319447); // Compare the shared vertex list and shared triangle list + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarSharedTriPath)); auto& exemplarDataArray = dataStructure.getDataRefAs(exemplarSharedTriPath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName))); auto& computedDataArray = dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName)); CompareDataArrays(exemplarDataArray, computedDataArray); CompareArrays(dataStructure, exemplarSharedVertexPath, computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedVertexListName)); @@ -316,6 +338,8 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding", "[SimplnxCore][SurfaceNetsF TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding Smoothing", "[SimplnxCore][SurfaceNetsFilter]") { UnitTest::LoadPlugins(); + bool forceOocAlgo = static_cast(GENERATE(from_range(nx::core::k_ForceOocTestValues))); + const nx::core::ForceOocAlgorithmGuard guard(forceOocAlgo); const nx::core::UnitTest::TestFileSentinel testDataSentinel(nx::core::unit_test::k_TestFilesDir, "SurfaceNetsTest_v3.tar.gz", "SurfaceNetsTest_v3"); @@ -343,6 +367,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding Smoothing", "[SimplnxCore][Su Arguments args; SurfaceNetsFilter const filter; + REQUIRE_NOTHROW(dataStructure.getDataRefAs(celDataPath)); auto voxelCellAttrMat = dataStructure.getDataRefAs(celDataPath); MultiArraySelectionParameter::ValueType selectedCellArrayPaths; for(const auto& child : voxelCellAttrMat) @@ -350,6 +375,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding Smoothing", "[SimplnxCore][Su selectedCellArrayPaths.push_back(celDataPath.createChildPath(child.second->getName())); } + REQUIRE_NOTHROW(dataStructure.getDataRefAs(featureDataPath)); auto voxelFeatureAttrMat = dataStructure.getDataRefAs(featureDataPath); MultiArraySelectionParameter::ValueType selectedFeatureArrayPaths; for(const auto& child : voxelFeatureAttrMat) @@ -390,6 +416,7 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding Smoothing", "[SimplnxCore][Su #endif } // Check a few things about the generated data. + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath)); TriangleGeom& triangleGeom = dataStructure.getDataRefAs(computedTriangleGeomPath); IGeometry::SharedTriList* triangle = triangleGeom.getFaces(); IGeometry::SharedVertexList* vertices = triangleGeom.getVertices(); @@ -398,7 +425,9 @@ TEST_CASE("SimplnxCore::SurfaceNetsFilter: Winding Smoothing", "[SimplnxCore][Su REQUIRE(vertices->getNumberOfTuples() == 319447); // Compare the shared vertex list and shared triangle list + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarSharedTriPath)); auto& exemplarDataArray = dataStructure.getDataRefAs(exemplarSharedTriPath); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName))); auto& computedDataArray = dataStructure.getDataRefAs(computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedFacesListName)); CompareDataArrays(exemplarDataArray, computedDataArray); CompareArrays(dataStructure, exemplarSharedVertexPath, computedTriangleGeomPath.createChildPath(TriangleGeom::k_SharedVertexListName)); diff --git a/src/simplnx/Utilities/AlignSections.cpp b/src/simplnx/Utilities/AlignSections.cpp index de5988f7a4..42680a4fd5 100644 --- a/src/simplnx/Utilities/AlignSections.cpp +++ b/src/simplnx/Utilities/AlignSections.cpp @@ -1,6 +1,7 @@ #include "AlignSections.hpp" #include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Utilities/AlgorithmDispatch.hpp" #include "simplnx/Utilities/DataGroupUtilities.hpp" #include "simplnx/Utilities/ParallelAlgorithmUtilities.hpp" #include "simplnx/Utilities/ParallelDataAlgorithm.hpp" @@ -14,6 +15,9 @@ using namespace nx::core; namespace { // ----------------------------------------------------------------------------- +// In-core transfer: original per-tuple copyTuple/initializeTuple with +// direction-dependent iteration to avoid overwriting source data. +// ----------------------------------------------------------------------------- template class AlignSectionsTransferDataImpl { @@ -92,6 +96,100 @@ class AlignSectionsTransferDataImpl } } +private: + AlignSections* m_Filter = nullptr; + SizeVec3 m_Dims; + std::vector m_Xshifts; + std::vector m_Yshifts; + nx::core::DataArray& m_DataArray; +}; + +// ----------------------------------------------------------------------------- +// OOC transfer: slice-buffered approach. Reads each Z-slice into a local buffer, +// applies the 2D X/Y shift in memory, then writes the shifted data back. +// All DataStore access is sequential, eliminating per-tuple chunk thrashing. +// ----------------------------------------------------------------------------- +template +class AlignSectionsTransferDataOocImpl +{ +public: + AlignSectionsTransferDataOocImpl() = delete; + AlignSectionsTransferDataOocImpl(const AlignSectionsTransferDataOocImpl&) = default; + AlignSectionsTransferDataOocImpl(AlignSectionsTransferDataOocImpl&&) noexcept = default; + + AlignSectionsTransferDataOocImpl(AlignSections* filter, SizeVec3 dims, std::vector xShifts, std::vector yShifts, IDataArray& dataArray) + : m_Filter(filter) + , m_Dims(std::move(dims)) + , m_Xshifts(std::move(xShifts)) + , m_Yshifts(std::move(yShifts)) + , m_DataArray(static_cast&>(dataArray)) + { + } + + ~AlignSectionsTransferDataOocImpl() = default; + + AlignSectionsTransferDataOocImpl& operator=(const AlignSectionsTransferDataOocImpl&) = delete; + AlignSectionsTransferDataOocImpl& operator=(AlignSectionsTransferDataOocImpl&&) = delete; + + void operator()() const + { + MessageHelper& messageHelper = m_Filter->getMessageHelper(); + ThrottledMessenger progressMessenger = messageHelper.createThrottledMessenger(); + + auto& dataStore = m_DataArray.getDataStoreRef(); + const usize numComp = m_DataArray.getNumberOfComponents(); + const usize dimX = m_Dims[0]; + const usize dimY = m_Dims[1]; + const usize sliceVoxels = dimX * dimY; + const usize sliceElements = sliceVoxels * numComp; + + std::string arrayName = m_DataArray.getName(); + + // Buffers for one Z-slice (use unique_ptr to avoid std::vector specialization) + auto sliceBuffer = std::make_unique(sliceElements); + auto outBuffer = std::make_unique(sliceElements); + + for(usize i = 1; i < m_Dims[2]; i++) + { + progressMessenger.sendThrottledMessage([&]() { return fmt::format("Processing {}: {:.2f}% completed", arrayName, CalculatePercentComplete(i, m_Dims[2])); }); + if(m_Filter->getCancel()) + { + return; + } + + usize slice = (m_Dims[2] - 1) - i; + usize sliceOffset = slice * sliceElements; + + // Phase 1: Bulk-read entire Z-slice into local buffer + dataStore.copyIntoBuffer(sliceOffset, nonstd::span(sliceBuffer.get(), sliceElements)); + + // Phase 2: Apply 2D X/Y shift in the output buffer, then bulk-write back + int64_t xShift = m_Xshifts[i]; + int64_t yShift = m_Yshifts[i]; + + std::fill(outBuffer.get(), outBuffer.get() + sliceElements, static_cast(0)); + for(usize yIndex = 0; yIndex < dimY; yIndex++) + { + for(usize xIndex = 0; xIndex < dimX; xIndex++) + { + int64_t srcX = static_cast(xIndex) + xShift; + int64_t srcY = static_cast(yIndex) + yShift; + + if(srcX >= 0 && srcX < static_cast(dimX) && srcY >= 0 && srcY < static_cast(dimY)) + { + usize srcBufBase = (static_cast(srcY) * dimX + static_cast(srcX)) * numComp; + usize dstBufBase = (yIndex * dimX + xIndex) * numComp; + for(usize c = 0; c < numComp; c++) + { + outBuffer[dstBufBase + c] = sliceBuffer[srcBufBase + c]; + } + } + } + } + dataStore.copyFromBuffer(sliceOffset, nonstd::span(outBuffer.get(), sliceElements)); + } + } + private: AlignSections* m_Filter = nullptr; SizeVec3 m_Dims; @@ -147,6 +245,21 @@ Result<> AlignSections::execute(const SizeVec3& udims, const DataPath& imageGeom // Now Adjust the actual DataArrays const std::vector selectedCellArrays = getSelectedDataPaths(imageGeometryPath); + // Determine whether to use OOC-optimized transfer + bool useOoc = ForceOocAlgorithm(); + if(!useOoc) + { + for(const auto& cellArrayPath : selectedCellArrays) + { + const auto& cellArray = m_DataStructure.getDataRefAs(cellArrayPath); + if(IsOutOfCore(cellArray)) + { + useOoc = true; + break; + } + } + } + ParallelTaskAlgorithm taskRunner; for(const auto& cellArrayPath : selectedCellArrays) @@ -158,7 +271,15 @@ Result<> AlignSections::execute(const SizeVec3& udims, const DataPath& imageGeom m_MessageHelper.sendMessage(fmt::format("Updating DataArray '{}'", cellArrayPath.toString())); auto& cellArray = m_DataStructure.getDataRefAs(cellArrayPath); - ExecuteParallelFunction(cellArray.getDataType(), taskRunner, this, udims, xShifts, yShifts, cellArray); + + if(useOoc) + { + ExecuteParallelFunction(cellArray.getDataType(), taskRunner, this, udims, xShifts, yShifts, cellArray); + } + else + { + ExecuteParallelFunction(cellArray.getDataType(), taskRunner, this, udims, xShifts, yShifts, cellArray); + } } // This will spill over if the number of DataArrays to process does not divide evenly by the number of threads. diff --git a/src/simplnx/Utilities/ClusteringUtilities.cpp b/src/simplnx/Utilities/ClusteringUtilities.cpp index 7a902544e3..1c2e895ecf 100644 --- a/src/simplnx/Utilities/ClusteringUtilities.cpp +++ b/src/simplnx/Utilities/ClusteringUtilities.cpp @@ -2,6 +2,8 @@ #include "simplnx/Utilities/DataStoreUtilities.hpp" +#include + #include using namespace nx::core; @@ -40,12 +42,19 @@ void RandomizeFeatureIds(Int32AbstractDataStore& featureIdsStore, usize totalFea { std::vector randomIds = CreateRandomizedIdsList(totalFeatures); - // Now adjust all the Grain ID values for each Voxel - // instead of taking total points as an input just extract the size, so we don't walk off + // Chunked bulk I/O for OOC efficiency usize totalPoints = featureIdsStore.getSize(); - for(int64 i = 0; i < totalPoints; ++i) + constexpr usize k_ChunkSize = 65536; + std::vector chunkBuf(k_ChunkSize); + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) { - featureIdsStore[i] = randomIds[featureIdsStore[i]]; + usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(chunkBuf.data(), count)); + for(usize i = 0; i < count; i++) + { + chunkBuf[i] = randomIds[chunkBuf[i]]; + } + featureIdsStore.copyFromBuffer(offset, nonstd::span(chunkBuf.data(), count)); } } @@ -53,17 +62,24 @@ void RandomizeFeatureIds(Int32AbstractDataStore& featureIdsStore, usize totalFea { std::vector randomIds = CreateRandomizedIdsList(totalFeatures); - // Now adjust all the Grain ID values for each Voxel - // instead of taking total points as an input just extract the size, so we don't walk off + // Chunked bulk I/O for OOC efficiency usize totalPoints = featureIdsStore.getSize(); - for(int64 i = 0; i < totalPoints; ++i) + constexpr usize k_ChunkSize = 65536; + std::vector chunkBuf(k_ChunkSize); + for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) { - featureIdsStore[i] = randomIds[featureIdsStore[i]]; + usize count = std::min(k_ChunkSize, totalPoints - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(chunkBuf.data(), count)); + for(usize i = 0; i < count; i++) + { + chunkBuf[i] = randomIds[chunkBuf[i]]; + } + featureIdsStore.copyFromBuffer(offset, nonstd::span(chunkBuf.data(), count)); } if(!featureIArrays.empty()) { - // We use a visitation pattern to prevent reverting swaps + // Visitation pattern for feature-level tuple swaps (small, no OOC concern) std::vector visited(randomIds.size(), false); for(usize i = 0; i < randomIds.size(); i++) { diff --git a/src/simplnx/Utilities/DataArrayUtilities.cpp b/src/simplnx/Utilities/DataArrayUtilities.cpp index 24c3349962..2e0fac087a 100644 --- a/src/simplnx/Utilities/DataArrayUtilities.cpp +++ b/src/simplnx/Utilities/DataArrayUtilities.cpp @@ -12,6 +12,8 @@ #include "simplnx/Utilities/StringInterpretationUtilities.hpp" #include "simplnx/Utilities/TemplateHelpers.hpp" +#include + using namespace nx::core; namespace @@ -179,7 +181,26 @@ Result<> ValidateFeatureIdsToFeatureAttributeMatrixIndexing(const DataStructure& } auto& featureIdsStore = featureIds.getDataStoreRef(); - auto [minFeatureId, maxFeatureId] = std::minmax_element(featureIdsStore.begin(), featureIdsStore.end()); + + // Use bulk I/O to find min/max instead of per-element iterators. + // Per-element iteration on OOC stores incurs ~50-100ns virtual dispatch per + // access; at 8M+ elements this takes minutes. Bulk copyIntoBuffer reads + // are a single HDF5 hyperslab op per batch. + const usize totalTuples = featureIdsStore.getNumberOfTuples(); + constexpr usize k_BatchSize = 40000; // roughly one 200x200 Z-slice + int32 globalMin = std::numeric_limits::max(); + int32 globalMax = std::numeric_limits::lowest(); + std::vector batch(k_BatchSize); + for(usize offset = 0; offset < totalTuples; offset += k_BatchSize) + { + const usize count = std::min(k_BatchSize, totalTuples - offset); + featureIdsStore.copyIntoBuffer(offset, nonstd::span(batch.data(), count)); + auto [batchMin, batchMax] = std::minmax_element(batch.begin(), batch.begin() + count); + globalMin = std::min(globalMin, *batchMin); + globalMax = std::max(globalMax, *batchMax); + } + const int32* minFeatureId = &globalMin; + const int32* maxFeatureId = &globalMax; if(!ignoreNegativeValues && *minFeatureId < 0) { diff --git a/src/simplnx/Utilities/DataArrayUtilities.hpp b/src/simplnx/Utilities/DataArrayUtilities.hpp index 3be14ebab7..d6a4e4818d 100644 --- a/src/simplnx/Utilities/DataArrayUtilities.hpp +++ b/src/simplnx/Utilities/DataArrayUtilities.hpp @@ -253,10 +253,7 @@ Result<> ImportFromBinaryFile(const std::filesystem::path& binaryFilePath, DataA return MakeErrorResult(-1001, fmt::format("Unexpected end of file or read error after reading {} of {} elements from '{}'", elementCounter, numElements, binaryFilePath.string())); } - for(usize i = 0; i < elementsRead; i++) - { - outputDataArray[i + elementCounter] = buffer[i]; - } + outputDataArray.getDataStoreRef().copyFromBuffer(elementCounter, nonstd::span(buffer.data(), elementsRead)); elementCounter += elementsRead; @@ -591,9 +588,27 @@ template void AppendData(const K& inputArray, K& destArray, usize offset) { const usize numElements = inputArray.getNumberOfTuples() * inputArray.getNumberOfComponents(); - for(usize i = 0; i < numElements; ++i) + if constexpr(requires { inputArray.getDataStoreRef(); }) + { + // DataArray path: use bulk I/O for OOC efficiency + using ValueType = typename std::remove_reference_t::value_type; + constexpr usize k_ChunkSize = 65536; + auto buffer = std::make_unique(std::min(numElements, k_ChunkSize)); + const auto& srcStore = inputArray.getDataStoreRef(); + auto& dstStore = destArray.getDataStoreRef(); + for(usize i = 0; i < numElements; i += k_ChunkSize) + { + usize count = std::min(k_ChunkSize, numElements - i); + srcStore.copyIntoBuffer(i, nonstd::span(buffer.get(), count)); + dstStore.copyFromBuffer(offset + i, nonstd::span(buffer.get(), count)); + } + } + else { - destArray.setValue(offset + i, inputArray.at(i)); + for(usize i = 0; i < numElements; ++i) + { + destArray.setValue(offset + i, inputArray.at(i)); + } } } @@ -625,10 +640,42 @@ Result<> CopyData(const K& inputArray, K& destArray, usize destTupleOffset, usiz return MakeErrorResult(-2034, fmt::format("The total number of elements to copy ({}) is larger than the total available elements ({}).", elementsToCopy, availableElements)); } - auto srcBegin = inputArray.begin() + (srcTupleOffset * sourceNumComponents); - auto srcEnd = srcBegin + (totalSrcTuples * sourceNumComponents); - auto dstBegin = destArray.begin() + (destTupleOffset * numComponents); - std::copy(srcBegin, srcEnd, dstBegin); + const usize numElements = totalSrcTuples * sourceNumComponents; + if constexpr(requires { inputArray.getDataStoreRef(); }) + { + const auto& srcStore = inputArray.getDataStoreRef(); + auto& dstStore = destArray.getDataStoreRef(); + if(srcStore.getStoreType() == IDataStore::StoreType::OutOfCore || dstStore.getStoreType() == IDataStore::StoreType::OutOfCore) + { + // OOC path: chunked bulk I/O + using ValueType = typename std::remove_reference_t::value_type; + constexpr usize k_ChunkSize = 65536; + auto buffer = std::make_unique(std::min(numElements, k_ChunkSize)); + usize srcStart = srcTupleOffset * sourceNumComponents; + usize dstStart = destTupleOffset * numComponents; + for(usize offset = 0; offset < numElements; offset += k_ChunkSize) + { + usize count = std::min(k_ChunkSize, numElements - offset); + srcStore.copyIntoBuffer(srcStart + offset, nonstd::span(buffer.get(), count)); + dstStore.copyFromBuffer(dstStart + offset, nonstd::span(buffer.get(), count)); + } + } + else + { + // In-core path: single std::copy (zero overhead) + auto srcBegin = inputArray.begin() + (srcTupleOffset * sourceNumComponents); + auto srcEnd = srcBegin + numElements; + auto dstBegin = destArray.begin() + (destTupleOffset * numComponents); + std::copy(srcBegin, srcEnd, dstBegin); + } + } + else + { + auto srcBegin = inputArray.begin() + (srcTupleOffset * sourceNumComponents); + auto srcEnd = srcBegin + numElements; + auto dstBegin = destArray.begin() + (destTupleOffset * numComponents); + std::copy(srcBegin, srcEnd, dstBegin); + } return {}; } @@ -661,10 +708,24 @@ Result<> CopyData(const std::vector& src, K& dst, usize dstTupleOffset, usize return MakeErrorResult(-2034, fmt::format("The total number of elements to copy ({}) is larger than the total available elements ({}).", elementsToCopy, dstAvailableElems)); } - auto srcBegin = src.begin() + (srcTupleOffset * srcNumComponents); - auto srcEnd = srcBegin + (totalSrcTuples * srcNumComponents); - auto dstBegin = dst.begin() + (dstTupleOffset * dstNumComponents); - std::copy(srcBegin, srcEnd, dstBegin); + const usize numElements = totalSrcTuples * srcNumComponents; + usize srcStart = srcTupleOffset * srcNumComponents; + usize dstStart = dstTupleOffset * dstNumComponents; + if constexpr(requires { dst.getDataStoreRef(); }) + { + dst.getDataStoreRef().copyFromBuffer(dstStart, nonstd::span(src.data() + srcStart, numElements)); + } + else if constexpr(requires { dst.copyFromBuffer(dstStart, nonstd::span(src.data(), 1)); }) + { + dst.copyFromBuffer(dstStart, nonstd::span(src.data() + srcStart, numElements)); + } + else + { + auto srcBegin = src.begin() + srcStart; + auto srcEnd = srcBegin + numElements; + auto dstBegin = dst.begin() + dstStart; + std::copy(srcBegin, srcEnd, dstBegin); + } return {}; } @@ -875,12 +936,49 @@ Result<> AppendDataX(const std::vector& inputArrays, const std::vector if(mirror) { auto numComps = destArray.getNumberOfComponents(); - for(usize x = 0; x < appendDestXDim / 2; ++x) + if constexpr(requires { destArray.getDataStoreRef(); }) { - usize tupleIdx = (z * appendYDim * appendDestXDim) + (y * appendDestXDim) + x; - usize endTupleIdx = tupleIdx + 1; - usize mirrorTupleIdx = (z * appendYDim * appendDestXDim) + (y * appendDestXDim) + (appendDestXDim - 1 - x); - std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + if(destArray.getDataStoreRef().getStoreType() == IDataStore::StoreType::OutOfCore) + { + // OOC path: read entire scanline, reverse tuples in-memory, write back + using ValueType = typename std::remove_reference_t::value_type; + usize scanlineElements = appendDestXDim * numComps; + auto scanline = std::make_unique(scanlineElements); + auto& store = destArray.getDataStoreRef(); + usize rowStart = ((z * appendYDim * appendDestXDim) + (y * appendDestXDim)) * numComps; + store.copyIntoBuffer(rowStart, nonstd::span(scanline.get(), scanlineElements)); + // Reverse tuple order in the buffer + for(usize x = 0; x < appendDestXDim / 2; ++x) + { + usize mirrorX = appendDestXDim - 1 - x; + for(usize c = 0; c < numComps; ++c) + { + std::swap(scanline[x * numComps + c], scanline[mirrorX * numComps + c]); + } + } + store.copyFromBuffer(rowStart, nonstd::span(scanline.get(), scanlineElements)); + } + else + { + // In-core path: original swap_ranges + for(usize x = 0; x < appendDestXDim / 2; ++x) + { + usize tupleIdx = (z * appendYDim * appendDestXDim) + (y * appendDestXDim) + x; + usize endTupleIdx = tupleIdx + 1; + usize mirrorTupleIdx = (z * appendYDim * appendDestXDim) + (y * appendDestXDim) + (appendDestXDim - 1 - x); + std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + } + } + } + else + { + for(usize x = 0; x < appendDestXDim / 2; ++x) + { + usize tupleIdx = (z * appendYDim * appendDestXDim) + (y * appendDestXDim) + x; + usize endTupleIdx = tupleIdx + 1; + usize mirrorTupleIdx = (z * appendYDim * appendDestXDim) + (y * appendDestXDim) + (appendDestXDim - 1 - x); + std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + } } } } @@ -926,16 +1024,60 @@ Result<> AppendDataY(const std::vector& inputArrays, const std::vector if(mirror) { auto numComps = destArray.getNumberOfComponents(); - for(int z = 0; z < appendZDim; ++z) + if constexpr(requires { destArray.getDataStoreRef(); }) { - for(int x = 0; x < appendXDim; ++x) + if(destArray.getDataStoreRef().getStoreType() == IDataStore::StoreType::OutOfCore) { - for(int y = 0; y < appendDestYDim / 2; ++y) + // OOC path: swap entire rows (scanlines) at once to minimize I/O calls. + // Instead of per-tuple swaps (xDim * yDim/2 * zDim calls), this does + // yDim/2 * zDim row-sized bulk reads/writes. + using ValueType = typename std::remove_reference_t::value_type; + usize rowElements = static_cast(appendXDim) * numComps; + auto rowA = std::make_unique(rowElements); + auto rowB = std::make_unique(rowElements); + auto& store = destArray.getDataStoreRef(); + for(int64 z = 0; z < appendZDim; ++z) { - usize tupleIdx = (z * appendDestYDim * appendXDim) + (y * appendXDim) + x; - usize endTupleIdx = tupleIdx + 1; - usize mirrorTupleIdx = (z * appendDestYDim * appendXDim) + ((appendDestYDim - 1 - y) * appendXDim) + x; - std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + for(usize y = 0; y < appendDestYDim / 2; ++y) + { + usize mirrorY = appendDestYDim - 1 - y; + usize offsetA = (static_cast(z) * appendDestYDim * static_cast(appendXDim) + y * static_cast(appendXDim)) * numComps; + usize offsetB = (static_cast(z) * appendDestYDim * static_cast(appendXDim) + mirrorY * static_cast(appendXDim)) * numComps; + store.copyIntoBuffer(offsetA, nonstd::span(rowA.get(), rowElements)); + store.copyIntoBuffer(offsetB, nonstd::span(rowB.get(), rowElements)); + store.copyFromBuffer(offsetA, nonstd::span(rowB.get(), rowElements)); + store.copyFromBuffer(offsetB, nonstd::span(rowA.get(), rowElements)); + } + } + } + else + { + // In-core path: swap entire rows using swap_ranges for efficiency + for(int64 z = 0; z < appendZDim; ++z) + { + for(usize y = 0; y < appendDestYDim / 2; ++y) + { + usize mirrorY = appendDestYDim - 1 - y; + usize rowIdx = static_cast(z) * appendDestYDim * static_cast(appendXDim) + y * static_cast(appendXDim); + usize mirrorRowIdx = static_cast(z) * appendDestYDim * static_cast(appendXDim) + mirrorY * static_cast(appendXDim); + usize rowElements = static_cast(appendXDim) * numComps; + std::swap_ranges(destArray.begin() + (rowIdx * numComps), destArray.begin() + (rowIdx * numComps + rowElements), destArray.begin() + (mirrorRowIdx * numComps)); + } + } + } + } + else + { + // Non-DataArray path: swap entire rows + for(int64 z = 0; z < appendZDim; ++z) + { + for(usize y = 0; y < appendDestYDim / 2; ++y) + { + usize mirrorY = appendDestYDim - 1 - y; + usize rowIdx = static_cast(z) * appendDestYDim * static_cast(appendXDim) + y * static_cast(appendXDim); + usize mirrorRowIdx = static_cast(z) * appendDestYDim * static_cast(appendXDim) + mirrorY * static_cast(appendXDim); + usize rowElements = static_cast(appendXDim) * numComps; + std::swap_ranges(destArray.begin() + (rowIdx * numComps), destArray.begin() + (rowIdx * numComps + rowElements), destArray.begin() + (mirrorRowIdx * numComps)); } } } @@ -967,12 +1109,47 @@ Result<> AppendDataZ(const std::vector& inputArrays, const std::vector auto appendDestZDim = newDestDims[0]; auto sliceTupleCount = newDestDims[1] * newDestDims[2]; auto numComps = destArray.getNumberOfComponents(); - for(int i = 0; i < appendDestZDim / 2; ++i) + if constexpr(requires { destArray.getDataStoreRef(); }) + { + if(destArray.getDataStoreRef().getStoreType() == IDataStore::StoreType::OutOfCore) + { + // OOC path: bulk I/O swap of entire Z-slices + using ValueType = typename std::remove_reference_t::value_type; + usize sliceElements = sliceTupleCount * numComps; + auto bufA = std::make_unique(sliceElements); + auto bufB = std::make_unique(sliceElements); + auto& store = destArray.getDataStoreRef(); + for(usize i = 0; i < appendDestZDim / 2; ++i) + { + usize offsetA = i * sliceElements; + usize offsetB = (appendDestZDim - 1 - i) * sliceElements; + store.copyIntoBuffer(offsetA, nonstd::span(bufA.get(), sliceElements)); + store.copyIntoBuffer(offsetB, nonstd::span(bufB.get(), sliceElements)); + store.copyFromBuffer(offsetA, nonstd::span(bufB.get(), sliceElements)); + store.copyFromBuffer(offsetB, nonstd::span(bufA.get(), sliceElements)); + } + } + else + { + // In-core path: original swap_ranges + for(int i = 0; i < appendDestZDim / 2; ++i) + { + usize tupleIdx = i * sliceTupleCount; + usize endTupleIdx = tupleIdx + sliceTupleCount; + usize mirrorTupleIdx = (appendDestZDim - 1 - i) * sliceTupleCount; + std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + } + } + } + else { - usize tupleIdx = i * sliceTupleCount; - usize endTupleIdx = tupleIdx + sliceTupleCount; - usize mirrorTupleIdx = (appendDestZDim - 1 - i) * sliceTupleCount; - std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + for(int i = 0; i < appendDestZDim / 2; ++i) + { + usize tupleIdx = i * sliceTupleCount; + usize endTupleIdx = tupleIdx + sliceTupleCount; + usize mirrorTupleIdx = (appendDestZDim - 1 - i) * sliceTupleCount; + std::swap_ranges(destArray.begin() + (tupleIdx * numComps), destArray.begin() + (endTupleIdx * numComps), destArray.begin() + (mirrorTupleIdx * numComps)); + } } } diff --git a/src/simplnx/Utilities/DataGroupUtilities.cpp b/src/simplnx/Utilities/DataGroupUtilities.cpp index 7ce17b4277..b27565f6ff 100644 --- a/src/simplnx/Utilities/DataGroupUtilities.cpp +++ b/src/simplnx/Utilities/DataGroupUtilities.cpp @@ -3,6 +3,10 @@ #include "simplnx/DataStructure/AttributeMatrix.hpp" #include "simplnx/DataStructure/BaseGroup.hpp" +#include + +#include + namespace nx::core { bool RemoveInactiveObjects(DataStructure& dataStructure, const DataPath& featureDataGroupPath, const std::vector& activeObjects, Int32AbstractDataStore& cellFeatureIds, @@ -79,20 +83,33 @@ bool RemoveInactiveObjects(DataStructure& dataStructure, const DataPath& feature // dataArray->getIDataStore()->resizeTuples(newShape); } - // Loop over all the points and correct all the feature names + // Renumber featureIds using chunked bulk I/O + constexpr size_t k_ChunkSize = 65536; size_t totalPoints = cellFeatureIds.getNumberOfTuples(); bool featureIdsChanged = false; - for(size_t i = 0; i < totalPoints; i++) + auto chunkBuf = std::make_unique(k_ChunkSize); + for(size_t offset = 0; offset < totalPoints; offset += k_ChunkSize) { - if(cellFeatureIds[i] >= 0 && cellFeatureIds[i] < newNames.size()) - { - cellFeatureIds.setValue(i, static_cast(newNames[cellFeatureIds[i]])); - featureIdsChanged = true; - } if(shouldCancel) { return false; } + size_t count = std::min(k_ChunkSize, totalPoints - offset); + cellFeatureIds.copyIntoBuffer(offset, nonstd::span(chunkBuf.get(), count)); + bool chunkModified = false; + for(size_t i = 0; i < count; i++) + { + if(chunkBuf[i] >= 0 && static_cast(chunkBuf[i]) < newNames.size()) + { + chunkBuf[i] = static_cast(newNames[chunkBuf[i]]); + chunkModified = true; + } + } + if(chunkModified) + { + cellFeatureIds.copyFromBuffer(offset, nonstd::span(chunkBuf.get(), count)); + featureIdsChanged = true; + } } if(featureIdsChanged) diff --git a/src/simplnx/Utilities/GeometryHelpers.hpp b/src/simplnx/Utilities/GeometryHelpers.hpp index 49de30a025..5eb03d8285 100644 --- a/src/simplnx/Utilities/GeometryHelpers.hpp +++ b/src/simplnx/Utilities/GeometryHelpers.hpp @@ -10,6 +10,7 @@ #include +#include #include #include @@ -193,36 +194,50 @@ usize FindNumEdges(const AbstractDataStore& faceStore, usize numVertices = (d template void FindElementsContainingVert(const DataArray* elemList, DynamicListArray* dynamicList, usize numVerts) { - auto& elems = *elemList; const usize numElems = elemList->getNumberOfTuples(); const usize numVertsPerElem = elemList->getNumberOfComponents(); + const auto& elemStore = elemList->getDataStoreRef(); // Allocate the basic structures std::vector linkCount(numVerts, 0); - - // Fill out lists with number of references to cells std::vector linkLoc(numVerts, static_cast(0)); - // vtkPolyData *pdata = static_cast(data); - // Traverse data to determine number of uses of each point - for(usize elemId = 0; elemId < numElems; elemId++) + // Use chunked bulk reads instead of per-element operator[] for OOC compatibility + constexpr usize k_ChunkElems = 65536; + auto chunkBuf = std::make_unique(k_ChunkElems * numVertsPerElem); + + // Chunked pass 1: count references to each vertex + for(usize start = 0; start < numElems; start += k_ChunkElems) { - usize offset = elemId * numVertsPerElem; - for(usize j = 0; j < numVertsPerElem; j++) + usize count = std::min(k_ChunkElems, numElems - start); + usize elemCount = count * numVertsPerElem; + elemStore.copyIntoBuffer(start * numVertsPerElem, nonstd::span(chunkBuf.get(), elemCount)); + for(usize i = 0; i < count; i++) { - ++linkCount[elems[offset + j]]; + for(usize j = 0; j < numVertsPerElem; j++) + { + ++linkCount[chunkBuf[i * numVertsPerElem + j]]; + } } } // Now allocate storage for the links dynamicList->allocateLists(linkCount); - for(usize elemId = 0; elemId < numElems; elemId++) + // Chunked pass 2: insert cell references + for(usize start = 0; start < numElems; start += k_ChunkElems) { - usize offset = elemId * numVertsPerElem; - for(usize j = 0; j < numVertsPerElem; j++) + usize count = std::min(k_ChunkElems, numElems - start); + usize elemCount = count * numVertsPerElem; + elemStore.copyIntoBuffer(start * numVertsPerElem, nonstd::span(chunkBuf.get(), elemCount)); + for(usize i = 0; i < count; i++) { - dynamicList->insertCellReference(elems[offset + j], (linkLoc[elems[offset + j]])++, elemId); + usize elemId = start + i; + for(usize j = 0; j < numVertsPerElem; j++) + { + K vertId = chunkBuf[i * numVertsPerElem + j]; + dynamicList->insertCellReference(vertId, (linkLoc[vertId])++, elemId); + } } } } @@ -240,9 +255,9 @@ void FindElementsContainingVert(const DataArray* elemList, DynamicListArray ErrorCode FindElementNeighbors(const DataArray* elemList, const DynamicListArray* elemsContainingVert, DynamicListArray* dynamicList, IGeometry::Type geometryType) { - auto& elems = *elemList; const usize numElems = elemList->getNumberOfTuples(); const usize numVertsPerElem = elemList->getNumberOfComponents(); + const auto& elemStore = elemList->getDataStoreRef(); usize numSharedVerts = 0; std::vector linkCount(numElems, 0); ErrorCode err = 0; @@ -292,68 +307,90 @@ ErrorCode FindElementNeighbors(const DataArray* elemList, const DynamicListAr // Reuse this vector for each loop. Avoids re-allocating the memory each time through the loop std::vector loop_neighbors(32, 0); - // Build up the element adjacency list now that we have the element links - for(usize t = 0; t < numElems; ++t) + // Buffer for reading a single candidate neighbor element's vertices via bulk I/O + auto neighborVertsBuf = std::make_unique(numVertsPerElem); + + // Process elements in chunks for the sequential outer read (OOC-friendly bulk I/O) + constexpr usize k_ChunkElems = 65536; + auto chunkBuf = std::make_unique(k_ChunkElems * numVertsPerElem); + + for(usize chunkStart = 0; chunkStart < numElems; chunkStart += k_ChunkElems) { - // qDebug() << "Analyzing Cell " << t << "\n"; - const usize offset = t * numVertsPerElem; - for(usize v = 0; v < numVertsPerElem; ++v) + usize chunkCount = std::min(k_ChunkElems, numElems - chunkStart); + elemStore.copyIntoBuffer(chunkStart * numVertsPerElem, nonstd::span(chunkBuf.get(), chunkCount * numVertsPerElem)); + + for(usize ci = 0; ci < chunkCount; ci++) { - // qDebug() << " vert " << v << "\n"; - T nEs = elemsContainingVert->getNumberOfElements(elems[offset + v]); - K* vertIdxs = elemsContainingVert->getElementListPointer(elems[offset + v]); + usize t = chunkStart + ci; + usize localOffset = ci * numVertsPerElem; - for(T vt = 0; vt < nEs; ++vt) + for(usize v = 0; v < numVertsPerElem; v++) { - if(vertIdxs[vt] == static_cast(t)) - { - continue; - } // This is the same element as our "source" - if(visited[vertIdxs[vt]]) - { - continue; - } // We already added this element so loop again - // qDebug() << " Comparing Element " << vertIdxs[vt] << "\n"; - auto vertCell = elemList->cbegin() + (vertIdxs[vt] * elemList->getNumberOfComponents()); - usize vCount = 0; - // Loop over all the vertex indices of this element and try to match numSharedVerts of them to the current loop element - // If there is numSharedVerts match then that element is a neighbor of the source. If there are more than numVertsPerElem - // matches then there is a real problem with the mesh and the program is going to return an error. - for(usize i = 0; i < numVertsPerElem; i++) + K vertId = chunkBuf[localOffset + v]; + T nEs = elemsContainingVert->getNumberOfElements(vertId); + K* vertIdxs = elemsContainingVert->getElementListPointer(vertId); + + for(T vt = 0; vt < nEs; vt++) { - for(usize j = 0; j < numVertsPerElem; j++) + if(vertIdxs[vt] == static_cast(t)) + { + continue; // This is the same element as our "source" + } + if(visited[vertIdxs[vt]]) { - if(elems[offset + i] == *(vertCell + j)) + continue; // We already added this element so loop again + } + + // Read the candidate neighbor element's vertices. + // If the candidate is within our current chunk, read from the local buffer; + // otherwise perform a single bulk read from the store. + K candidateElem = vertIdxs[vt]; + const K* candidateVerts = nullptr; + if(candidateElem >= chunkStart && candidateElem < chunkStart + chunkCount) + { + candidateVerts = &chunkBuf[(candidateElem - chunkStart) * numVertsPerElem]; + } + else + { + elemStore.copyIntoBuffer(candidateElem * numVertsPerElem, nonstd::span(neighborVertsBuf.get(), numVertsPerElem)); + candidateVerts = neighborVertsBuf.get(); + } + + // Count shared vertices between source element t and the candidate + usize vCount = 0; + for(usize i = 0; i < numVertsPerElem; i++) + { + for(usize j = 0; j < numVertsPerElem; j++) { - vCount++; + if(chunkBuf[localOffset + i] == candidateVerts[j]) + { + vCount++; + } } } - } - // So if our vertex match count is numSharedVerts, and we have not visited the element in question then add this element index - // into the list of vertex indices as neighbors for the source element. - if(vCount == numSharedVerts) - { - // qDebug() << " Neighbor: " << vertIdxs[vt] << "\n"; - // Use the current count of neighbors as the index - // into the loop_neighbors vector and place the value of the vertex element at that index - loop_neighbors[linkCount[t]] = vertIdxs[vt]; - ++linkCount[t]; // Increment the count for the next time through - if(linkCount[t] >= loop_neighbors.size()) + // If vertex match count equals numSharedVerts, this candidate is a neighbor + if(vCount == numSharedVerts) { - loop_neighbors.resize(loop_neighbors.size() + 10); + loop_neighbors[linkCount[t]] = vertIdxs[vt]; + ++linkCount[t]; + if(linkCount[t] >= loop_neighbors.size()) + { + loop_neighbors.resize(loop_neighbors.size() + 10); + } + visited[vertIdxs[vt]] = true; } - visited[vertIdxs[vt]] = true; // Set this element as visited so we do NOT add it again } } + + // Reset all the visited cell indices back to false (zero) + for(int64 k = 0; k < linkCount[t]; k++) + { + visited[loop_neighbors[k]] = false; + } + // Allocate the array storage for the current element to hold its neighbor list + dynamicList->setElementList(t, linkCount[t], &(loop_neighbors[0])); } - // Reset all the visited cell indexs back to false (zero) - for(int64 k = 0; k < linkCount[t]; ++k) - { - visited[loop_neighbors[k]] = false; - } - // Allocate the array storage for the current edge to hold its vertex list - dynamicList->setElementList(t, linkCount[t], &(loop_neighbors[0])); } return err; diff --git a/src/simplnx/Utilities/ImageRotationUtilities.hpp b/src/simplnx/Utilities/ImageRotationUtilities.hpp index 24e7bed348..6700019bd2 100644 --- a/src/simplnx/Utilities/ImageRotationUtilities.hpp +++ b/src/simplnx/Utilities/ImageRotationUtilities.hpp @@ -24,22 +24,22 @@ const Eigen::Vector3f k_XAxis = Eigen::Vector3f::UnitX(); const Eigen::Vector3f k_YAxis = Eigen::Vector3f::UnitY(); const Eigen::Vector3f k_ZAxis = Eigen::Vector3f::UnitZ(); -using Matrix3fR = Eigen::Matrix; -using Matrix4fR = Eigen::Matrix; +using Matrix3fR = Eigen::Matrix; +using Matrix4fR = Eigen::Matrix; -using Vector3i64 = Eigen::Array; +using Vector3i64 = Eigen::Array; struct RotateArgs { USizeVec3 OriginalDims; FloatVec3 OriginalSpacing; FloatVec3 OriginalOrigin; - int64_t xp = 0; - int64_t yp = 0; - int64_t zp = 0; - float xRes = 0.0f; - float yRes = 0.0f; - float zRes = 0.0f; + int64 xp = 0; + int64 yp = 0; + int64 zp = 0; + float32 xRes = 0.0f; + float32 yRes = 0.0f; + float32 zRes = 0.0f; USizeVec3 TransformedDims; FloatVec3 TransformedSpacing; @@ -48,9 +48,9 @@ struct RotateArgs USizeVec3 outputDims; FloatVec3 outputSpacing; - float outputXMin = 0.0f; - float outputYMin = 0.0f; - float outputZMin = 0.0f; + float32 outputXMin = 0.0f; + float32 outputYMin = 0.0f; + float32 outputZMin = 0.0f; }; /** @@ -138,7 +138,7 @@ T CosBetweenVectors(const Eigen::Vector3& vectorA, const Eigen::Vector3& v * @param axisNew * @return spacing for a given axis. */ -SIMPLNX_EXPORT float DetermineSpacing(const FloatVec3& spacing, const Eigen::Vector3f& axisNew); +SIMPLNX_EXPORT float32 DetermineSpacing(const FloatVec3& spacing, const Eigen::Vector3f& axisNew); /** * @brief Determines parameters for image rotation @@ -158,7 +158,7 @@ SIMPLNX_EXPORT ImageRotationUtilities::RotateArgs CreateRotationArgs(const Image * @return */ template -T inline GetSourceArrayValue(const RotateArgs& params, Vector3i64 xyzIndex, const DataArray& sourceArray, size_t compIndex) +T inline GetSourceArrayValue(const RotateArgs& params, Vector3i64 xyzIndex, const DataArray& sourceArray, usize compIndex) { if(xyzIndex[0] < 0) { @@ -199,7 +199,7 @@ T inline GetSourceArrayValue(const RotateArgs& params, Vector3i64 xyzIndex, cons * @param coord * @return */ -SIMPLNX_EXPORT size_t FindOctant(const RotateArgs& params, const Point3Df& centerPoint, const Eigen::Array4f& coord); +SIMPLNX_EXPORT usize FindOctant(const RotateArgs& params, const Point3Df& centerPoint, const Eigen::Array4f& coord); using OctantOffsetArrayType = std::array; @@ -241,20 +241,20 @@ using AccumulationValueType = std::conditional_t, fl * @param hitVoxelCenterPoint */ template -inline void FindInterpolationValues(const RotateArgs& params, size_t octant, SizeVec3 oldIndicesU, Eigen::Array4f& oldCoords, const DataArray& sourceArray, +inline void FindInterpolationValues(const RotateArgs& params, usize octant, SizeVec3 oldIndicesU, Eigen::Array4f& oldCoords, const DataArray& sourceArray, std::vector>& pValues, Eigen::Vector3f& uvw, Point3Df& hitVoxelCenterPoint) { const std::array& indexOffset = k_AllOctantOffsets[octant]; - const Vector3i64 oldIndices(static_cast(oldIndicesU[0]), static_cast(oldIndicesU[1]), static_cast(oldIndicesU[2])); - size_t numComps = sourceArray.getNumberOfComponents(); + const Vector3i64 oldIndices(static_cast(oldIndicesU[0]), static_cast(oldIndicesU[1]), static_cast(oldIndicesU[2])); + usize numComps = sourceArray.getNumberOfComponents(); Eigen::Vector3f p1Coord; - for(size_t i = 0; i < 8; i++) + for(usize i = 0; i < 8; i++) { auto pIndices = oldIndices + indexOffset[i]; - for(size_t compIndex = 0; compIndex < numComps; compIndex++) + for(usize compIndex = 0; compIndex < numComps; compIndex++) { T value = GetSourceArrayValue(params, pIndices, sourceArray, compIndex); pValues[i * numComps + compIndex] = value; @@ -277,7 +277,7 @@ inline void FindInterpolationValues(const RotateArgs& params, size_t octant, Siz static_cast(c111_Index[1]) * params.yRes + (0.5F * params.yRes) + params.OriginalOrigin[1], static_cast(c111_Index[2]) * params.zRes + (0.5F * params.zRes) + params.OriginalOrigin[2]}; - for(size_t i = 0; i < 3; i++) + for(usize i = 0; i < 3; i++) { uvw[i] = (oldCoords[i] - c000_Coord[i]) / (c111_Coord[i] - c000_Coord[i]); uvw[i] = uvw[i] < 0.0 ? 0.0 : uvw[i]; @@ -297,7 +297,7 @@ class FilterProgressCallback { } - void sendThreadSafeProgressMessage(int64_t counter) + void sendThreadSafeProgressMessage(int64 counter) { static std::mutex mutex; m_Progcounter += static_cast(counter); @@ -377,16 +377,16 @@ class RotateImageGeometryWithTrilinearInterpolation * @param indices * @return */ - T calculateInterpolatedValue(const std::vector>& pValues, const Eigen::Vector3f& uvw, size_t numComps, size_t compIndex) const + T calculateInterpolatedValue(const std::vector>& pValues, const Eigen::Vector3f& uvw, usize numComps, usize compIndex) const { - constexpr size_t P1 = 0; - constexpr size_t P2 = 1; - constexpr size_t P3 = 2; - constexpr size_t P4 = 3; - constexpr size_t P5 = 4; - constexpr size_t P6 = 5; - constexpr size_t P7 = 6; - constexpr size_t P8 = 7; + constexpr usize P1 = 0; + constexpr usize P2 = 1; + constexpr usize P3 = 2; + constexpr usize P4 = 3; + constexpr usize P5 = 4; + constexpr usize P6 = 5; + constexpr usize P7 = 6; + constexpr usize P8 = 7; /* clang-format on */ const AccumulationValueType c000 = pValues[P1 * numComps + compIndex]; @@ -398,9 +398,9 @@ class RotateImageGeometryWithTrilinearInterpolation const AccumulationValueType c111 = pValues[P7 * numComps + compIndex]; const AccumulationValueType c011 = pValues[P8 * numComps + compIndex]; - const float Xd = uvw[0]; - const float Yd = uvw[1]; - const float Zd = uvw[2]; + const float32 Xd = uvw[0]; + const float32 Yd = uvw[1]; + const float32 Zd = uvw[2]; const AccumulationValueType c00 = c000 * (1 - Xd) + c100 * Xd; const AccumulationValueType c01 = c001 * (1 - Xd) + c101 * Xd; @@ -425,7 +425,7 @@ class RotateImageGeometryWithTrilinearInterpolation using DataArrayType = DataArray; const auto& sourceArray = dynamic_cast(*m_SourceArray); - const size_t numComps = sourceArray.getNumberOfComponents(); + const usize numComps = sourceArray.getNumberOfComponents(); if(numComps == 0) { m_FilterCallback->sendThreadSafeProgressMessage(fmt::format("{}: Number of Components was Zero for array. Exiting Transform.", sourceArray.getName())); @@ -451,21 +451,21 @@ class RotateImageGeometryWithTrilinearInterpolation Matrix4fR inverseTransform = m_TransformationMatrix.inverse(); - for(int64_t k = 0; k < m_Params.outputDims[2]; k++) + for(int64 k = 0; k < m_Params.outputDims[2]; k++) { if(m_FilterCallback->getCancel()) { break; } m_FilterCallback->sendThreadSafeProgressMessage(fmt::format("{}: Interpolating values for slice '{}/{}'", m_SourceArray->getName(), k, m_Params.outputDims[2])); - int64_t ktot = (m_Params.outputDims[0] * m_Params.outputDims[1]) * k; + int64 ktot = (m_Params.outputDims[0] * m_Params.outputDims[1]) * k; - for(int64_t j = 0; j < m_Params.outputDims[1]; j++) + for(int64 j = 0; j < m_Params.outputDims[1]; j++) { - int64_t jtot = (m_Params.outputDims[0]) * j; - for(int64_t i = 0; i < m_Params.outputDims[0]; i++) + int64 jtot = (m_Params.outputDims[0]) * j; + for(int64 i = 0; i < m_Params.outputDims[0]; i++) { - int64_t destIndex = ktot + jtot + i; + int64 destIndex = ktot + jtot + i; Point3Df destPoint = destImageGeomPtr->getCoordsf(destIndex); // Last value is 1. See https://www.euclideanspace.com/maths/geometry/affine/matrix4x4/index.htm Eigen::Vector4f coordsNew(destPoint.getX(), destPoint.getY(), destPoint.getZ(), 1.0f); @@ -479,7 +479,7 @@ class RotateImageGeometryWithTrilinearInterpolation // Now we know what voxel the new cell center maps back to in the original geometry. if(errorResult == ImageGeom::ErrorType::NoError) { - size_t oldIndex = (m_Params.OriginalDims[0] * m_Params.OriginalDims[1] * oldGeomIndices[2]) + (m_Params.OriginalDims[0] * oldGeomIndices[1]) + oldGeomIndices[0]; + usize oldIndex = (m_Params.OriginalDims[0] * m_Params.OriginalDims[1] * oldGeomIndices[2]) + (m_Params.OriginalDims[0] * oldGeomIndices[1]) + oldGeomIndices[0]; auto oldVoxelCenterPoint = origImageGeomPtr->getCoordsf(oldIndex); @@ -488,7 +488,7 @@ class RotateImageGeometryWithTrilinearInterpolation Eigen::Vector3f uvw; FindInterpolationValues(m_Params, octant, oldGeomIndices, coordsOld, sourceArray, pValues, uvw, oldVoxelCenterPoint); - for(size_t compIndex = 0; compIndex < numComps; compIndex++) + for(usize compIndex = 0; compIndex < numComps; compIndex++) { T value = calculateInterpolatedValue(pValues, uvw, numComps, compIndex); newDataStore.setComponent(destIndex, compIndex, value); @@ -553,8 +553,25 @@ class RotateImageGeometryWithNearestNeighbor const auto& oldDataStore = m_SourceArray->template getIDataStoreRefAs>(); auto& newDataStore = m_TargetArray->template getIDataStoreRefAs>(); + const usize numComps = oldDataStore.getNumberOfComponents(); + const int64 srcDimX = static_cast(m_Params.OriginalDims[0]); + const int64 srcDimY = static_cast(m_Params.OriginalDims[1]); + const int64 srcDimZ = static_cast(m_Params.OriginalDims[2]); + const usize srcSliceSize = static_cast(srcDimX * srcDimY); + const usize outSliceSize = static_cast(m_Params.outputDims[0] * m_Params.outputDims[1]); Matrix4fR inverseTransform = m_TransformationMatrix.inverse(); + + // Allocate output slice buffer (bounded: one Z-slice of the output geometry) + auto outSliceBuf = std::make_unique(outSliceSize * numComps); + std::fill(outSliceBuf.get(), outSliceBuf.get() + outSliceSize * numComps, static_cast(0)); + + // Source slab cache: holds a contiguous range of source Z-slices + std::unique_ptr srcSlabBuf; + usize srcSlabBufSize = 0; + int64 cachedSrcZMin = -1; + int64 cachedSrcZMax = -2; // invalid range initially + for(int64 k = 0; k < m_Params.outputDims[2]; k++) { if(m_FilterCallback->getCancel()) @@ -563,46 +580,108 @@ class RotateImageGeometryWithNearestNeighbor } m_FilterCallback->sendThreadSafeProgressMessage(fmt::format("{}: Interpolating values for slice '{}/{}'", m_SourceArray->getName(), k, m_Params.outputDims[2])); - int64 const ktot = (m_Params.outputDims[0] * m_Params.outputDims[1]) * k; + // Determine source Z range needed for this output slice analytically. + // The inverse transform maps output physical coords to source physical coords. + // Source Z is a linear function of output (X, Y) for a fixed output Z, so + // extrema occur at the corners of the output slice's XY bounding box. + int64 neededZMin = srcDimZ; + int64 neededZMax = -1; + + if(m_SliceBySlice) + { + neededZMin = k; + neededZMax = k; + } + else + { + // Probe all 4 corners — compute source Z regardless of whether the point is in-bounds + for(int cj = 0; cj <= 1; cj++) + { + for(int ci = 0; ci <= 1; ci++) + { + int64 cx = ci == 0 ? 0 : static_cast(m_Params.outputDims[0] - 1); + int64 cy = cj == 0 ? 0 : static_cast(m_Params.outputDims[1] - 1); + int64 cornerFlatIdx = cx + cy * static_cast(m_Params.outputDims[0]) + k * static_cast(m_Params.outputDims[0] * m_Params.outputDims[1]); + Point3Df cornerPt = destImageGeomPtr->getCoordsf(cornerFlatIdx); + Eigen::Vector4f cornerNew(cornerPt.getX(), cornerPt.getY(), cornerPt.getZ(), 1.0f); + Eigen::Array4f cornerOld = inverseTransform * cornerNew; + // Convert source physical Z to cell index (floor division) + float srcPhysZ = cornerOld[2]; + float srcOriginZ = m_Params.OriginalOrigin[2]; + float srcSpacingZ = m_Params.OriginalSpacing[2]; + int64 srcZIdx = static_cast(std::floor((srcPhysZ - srcOriginZ) / srcSpacingZ)); + neededZMin = std::min(neededZMin, srcZIdx); + neededZMax = std::max(neededZMax, srcZIdx); + } + } + // Clamp to valid source range with margin + neededZMin = std::max(static_cast(0), neededZMin - 1); + neededZMax = std::min(srcDimZ - 1, neededZMax + 1); + } + + if(neededZMin > neededZMax || neededZMin >= srcDimZ || neededZMax < 0) + { + // No valid source mapping for this slice — fill with zeros + std::fill(outSliceBuf.get(), outSliceBuf.get() + outSliceSize * numComps, static_cast(0)); + newDataStore.copyFromBuffer(static_cast(k) * outSliceSize * numComps, nonstd::span(outSliceBuf.get(), outSliceSize * numComps)); + continue; + } + neededZMin = std::max(neededZMin, static_cast(0)); + neededZMax = std::min(neededZMax, srcDimZ - 1); + + // Read source slab if not already cached (or if range changed) + if(neededZMin < cachedSrcZMin || neededZMax > cachedSrcZMax) + { + cachedSrcZMin = neededZMin; + cachedSrcZMax = neededZMax; + usize slabTuples = static_cast(cachedSrcZMax - cachedSrcZMin + 1) * srcSliceSize; + usize slabElements = slabTuples * numComps; + if(slabElements > srcSlabBufSize) + { + srcSlabBuf = std::make_unique(slabElements); + srcSlabBufSize = slabElements; + } + oldDataStore.copyIntoBuffer(static_cast(cachedSrcZMin) * srcSliceSize * numComps, nonstd::span(srcSlabBuf.get(), slabElements)); + } + + // Process output slice + std::fill(outSliceBuf.get(), outSliceBuf.get() + outSliceSize * numComps, static_cast(0)); + + int64 const ktot = static_cast(m_Params.outputDims[0] * m_Params.outputDims[1]) * k; for(int64 j = 0; j < m_Params.outputDims[1]; j++) { - int64 jtot = (m_Params.outputDims[0]) * j; + int64 jtot = static_cast(m_Params.outputDims[0]) * j; for(int64 i = 0; i < m_Params.outputDims[0]; i++) { const int64 destIndex = ktot + jtot + i; + const usize outBufIdx = static_cast(j * static_cast(m_Params.outputDims[0]) + i); Point3Df destPoint = destImageGeomPtr->getCoordsf(destIndex); - // Last value is 1. See https://www.euclideanspace.com/maths/geometry/affine/matrix4x4/index.htm Eigen::Vector4f coordsNew(destPoint.getX(), destPoint.getY(), destPoint.getZ(), 1.0f); - // Transform back to the old coordinate Eigen::Array4f coordsOld = inverseTransform * coordsNew; - // Now compute the old Cell Index from the old coordinate SizeVec3 oldGeomIndices; auto errorResult = srcImageGeomPtr->computeCellIndex(coordsOld.data(), oldGeomIndices); - // Now we know what voxel the new cell center maps back to in the original geometry. if(errorResult == ImageGeom::ErrorType::NoError) { if(m_SliceBySlice) { oldGeomIndices[2] = k; } - size_t oldIndex = (m_Params.OriginalDims[0] * m_Params.OriginalDims[1] * oldGeomIndices[2]) + (m_Params.OriginalDims[0] * oldGeomIndices[1]) + oldGeomIndices[0]; - - if(newDataStore.copyFrom(destIndex, oldDataStore, oldIndex, 1).invalid()) + int64 srcZ = static_cast(oldGeomIndices[2]); + if(srcZ >= cachedSrcZMin && srcZ <= cachedSrcZMax) { - std::cout << fmt::format("Array copy failed: Source Array Name: {} Source Tuple Index: {}\nDest Array Name: {} Dest. Tuple Index {}\n", m_SourceArray->getName(), oldIndex, - m_SourceArray->getName(), destIndex) - << std::endl; - break; + usize slabLocalIdx = (static_cast(srcZ - cachedSrcZMin) * srcSliceSize + oldGeomIndices[1] * static_cast(srcDimX) + oldGeomIndices[0]) * numComps; + for(usize c = 0; c < numComps; c++) + { + outSliceBuf.get()[outBufIdx * numComps + c] = srcSlabBuf.get()[slabLocalIdx + c]; + } } } - else - { - newDataStore.fillTuple(destIndex, 0); - } } } + + newDataStore.copyFromBuffer(static_cast(k) * outSliceSize * numComps, nonstd::span(outSliceBuf.get(), outSliceSize * numComps)); } m_FilterCallback->sendThreadSafeProgressMessage(fmt::format("{}: Transform Ending", m_SourceArray->getName())); } @@ -634,13 +713,13 @@ class ApplyTransformationToNodeGeometry { } - void convert(size_t start, size_t end) const + void convert(usize start, usize end) const { - int64_t progCounter = 0; - const size_t totalElements = (end - start); - const size_t progIncrement = static_cast(totalElements / 100); + int64 progCounter = 0; + const usize totalElements = (end - start); + const usize progIncrement = static_cast(totalElements / 100); - for(size_t i = start; i < end; i++) + for(usize i = start; i < end; i++) { if(m_FilterCallback->getCancel()) { diff --git a/src/simplnx/Utilities/Meshing/TriangleUtilities.cpp b/src/simplnx/Utilities/Meshing/TriangleUtilities.cpp index 3dacace263..fb2a28fafd 100644 --- a/src/simplnx/Utilities/Meshing/TriangleUtilities.cpp +++ b/src/simplnx/Utilities/Meshing/TriangleUtilities.cpp @@ -3,8 +3,11 @@ #include #include +#include + #include #include +#include #include using namespace nx::core; @@ -13,8 +16,8 @@ namespace { using EdgeListT = std::set>; -Result<> ProcessWindingsWithLabels(INodeGeometry2D::SharedFaceList::store_type& triangles, const DynamicListArray& neighbors, - const Int32AbstractDataStore& faceLabelsStore, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler, int32 maxFeature) +Result<> ProcessWindingsWithLabels(IGeometry::MeshIndexType* triangles, usize numTris, const DynamicListArray& neighbors, const int32* faceLabels, + const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler, int32 maxFeature) { /** * This works by making a map of the edges since a properly wound mesh @@ -31,17 +34,16 @@ Result<> ProcessWindingsWithLabels(INodeGeometry2D::SharedFaceList::store_type& // Walk the features repairing the graph group by group usize count = 0; auto start = std::chrono::steady_clock::now(); - const usize numTuples = faceLabelsStore.getNumberOfTuples(); - std::vector visited(faceLabelsStore.getNumberOfTuples(), false); - std::vector unmodified(faceLabelsStore.getNumberOfTuples(), false); + std::vector visited(numTris, false); + std::vector unmodified(numTris, false); for(int32 feature = 1; feature < maxFeature + 1; feature++) { std::queue searchTargets = {}; // process base case - for(usize i = 0; i < numTuples; i++) + for(usize i = 0; i < numTris; i++) { - if(faceLabelsStore[i * 2] != feature && faceLabelsStore[(i * 2) + 1] != feature) + if(faceLabels[i * 2] != feature && faceLabels[(i * 2) + 1] != feature) { continue; } @@ -52,7 +54,7 @@ Result<> ProcessWindingsWithLabels(INodeGeometry2D::SharedFaceList::store_type& for(uint16 element = 0; element < numElem; element++) { const usize neighbor = neighborListPtr[element]; - if(faceLabelsStore[neighbor * 2] != feature && faceLabelsStore[(neighbor * 2) + 1] != feature) + if(faceLabels[neighbor * 2] != feature && faceLabels[(neighbor * 2) + 1] != feature) { continue; } @@ -94,7 +96,7 @@ Result<> ProcessWindingsWithLabels(INodeGeometry2D::SharedFaceList::store_type& for(uint16 element = 0; element < numElem; element++) { const usize neighbor = neighborListPtr[element]; - if(faceLabelsStore[neighbor * 2] != feature && faceLabelsStore[(neighbor * 2) + 1] != feature) + if(faceLabels[neighbor * 2] != feature && faceLabels[(neighbor * 2) + 1] != feature) { continue; } @@ -138,8 +140,8 @@ Result<> ProcessWindingsWithLabels(INodeGeometry2D::SharedFaceList::store_type& edgeList.find(std::make_pair(triangles[(triangle * 3) + 2], triangles[(triangle * 3) + 0])) != edgeList.end()) // If true it contains a conflicting edge { // check if previously visited - const usize offset = faceLabelsStore[triangle * 2] == feature ? 1 : 0; - const int32 alternateLabel = faceLabelsStore[(triangle * 2) + offset]; + const usize offset = faceLabels[triangle * 2] == feature ? 1 : 0; + const int32 alternateLabel = faceLabels[(triangle * 2) + offset]; if(alternateLabel != 0 && alternateLabel < feature) { unmodified[triangle] = true; @@ -164,8 +166,8 @@ Result<> ProcessWindingsWithLabels(INodeGeometry2D::SharedFaceList::store_type& return {}; } -Result<> ProcessWindingsWithRegions(INodeGeometry2D::SharedFaceList::store_type& triangles, const DynamicListArray& neighbors, - const Int32AbstractDataStore& regionsStore, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler, int32 maxFeature) +Result<> ProcessWindingsWithRegions(IGeometry::MeshIndexType* triangles, usize numTris, const DynamicListArray& neighbors, const int32* regions, + const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler, int32 maxFeature) { /** * This works by making a map of the edges since a properly wound mesh @@ -181,16 +183,15 @@ Result<> ProcessWindingsWithRegions(INodeGeometry2D::SharedFaceList::store_type& // Walk the features repairing the graph group by group auto start = std::chrono::steady_clock::now(); - const usize numTuples = regionsStore.getNumberOfTuples(); - std::vector visited(regionsStore.getNumberOfTuples(), false); + std::vector visited(numTris, false); for(int32 feature = 1; feature < maxFeature + 1; feature++) { std::queue searchTargets = {}; // process base case - for(usize i = 0; i < numTuples; i++) + for(usize i = 0; i < numTris; i++) { - if(regionsStore[i] != feature) + if(regions[i] != feature) { continue; } @@ -201,7 +202,7 @@ Result<> ProcessWindingsWithRegions(INodeGeometry2D::SharedFaceList::store_type& for(uint16 element = 0; element < numElem; element++) { const usize neighbor = neighborListPtr[element]; - if(regionsStore[neighbor] != feature) + if(regions[neighbor] != feature) { continue; } @@ -243,7 +244,7 @@ Result<> ProcessWindingsWithRegions(INodeGeometry2D::SharedFaceList::store_type& for(uint16 element = 0; element < numElem; element++) { const usize neighbor = neighborListPtr[element]; - if(regionsStore[neighbor] != feature) + if(regions[neighbor] != feature) { continue; } @@ -318,20 +319,38 @@ Result<> MeshingUtilities::RepairTriangleWinding(INodeGeometry2D::SharedFaceList fmt::format("MeshingUtilities::RepairTriangleWinding: invalid ID array supplied. The ID array must have 1 or 2 components, supplied array components: {}.", numComp)); } - // Get max group and (feature id != 0) + const usize numTris = triangles.getNumberOfTuples(); + const usize idsSize = idsStore.getSize(); // numTris * numComp + + // Bulk-read triangles into local buffer to avoid per-element OOC overhead + auto triBuf = std::make_unique(numTris * 3); + triangles.copyIntoBuffer(0, nonstd::span(triBuf.get(), numTris * 3)); + + // Bulk-read ids into local buffer + auto idsBuf = std::make_unique(idsSize); + idsStore.copyIntoBuffer(0, nonstd::span(idsBuf.get(), idsSize)); + + // Find max feature from local buffer int32 maxFeature = 0; - for(int32 i = 0; i < idsStore.getSize(); i++) + for(usize i = 0; i < idsSize; i++) { - maxFeature = std::max(idsStore[i], maxFeature); + maxFeature = std::max(idsBuf[i], maxFeature); } + Result<> result; if(numComp == 2) { - return ::ProcessWindingsWithLabels(triangles, neighbors, idsStore, shouldCancel, mesgHandler, maxFeature); + result = ::ProcessWindingsWithLabels(triBuf.get(), numTris, neighbors, idsBuf.get(), shouldCancel, mesgHandler, maxFeature); } + else + { + result = ::ProcessWindingsWithRegions(triBuf.get(), numTris, neighbors, idsBuf.get(), shouldCancel, mesgHandler, maxFeature); + } + + // Bulk-write modified triangles back + triangles.copyFromBuffer(0, nonstd::span(triBuf.get(), numTris * 3)); - // numComp == 1 - return ::ProcessWindingsWithRegions(triangles, neighbors, idsStore, shouldCancel, mesgHandler, maxFeature); + return result; } MeshingUtilities::CalculateNormalsImpl::CalculateNormalsImpl(const INodeGeometry2D::SharedFaceList::store_type& triangles, const INodeGeometry2D::SharedVertexList::store_type& verts, diff --git a/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp b/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp index dbf9617c7e..359a133b05 100644 --- a/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp +++ b/src/simplnx/Utilities/Parsing/HDF5/H5DataStore.hpp @@ -47,32 +47,84 @@ template Result<> FillOocDataStore(DataArray& dataArray, const DataPath& dataArrayPath, const nx::core::HDF5::DatasetIO& datasetReader, const std::optional>& start = std::nullopt, const std::optional>& count = std::nullopt) { - if(Memory::GetTotalMemory() <= dataArray.getSize() * sizeof(T)) + auto& absDataStore = dataArray.getDataStoreRef(); + + // Streaming path: read HDF5 dataset in row-batches using hyperslab reads + // to avoid allocating a buffer proportional to the full dataset size. + // When start/count are provided, the read is restricted to that sub-region. + auto dims = datasetReader.getDimensions(); + if(dims.empty()) { - return MakeErrorResult(-21004, fmt::format("Error reading dataset '{}' with '{}' total elements. Not enough memory to import data.", dataArray.getName(), datasetReader.getNumElements())); + return MakeErrorResult(-21005, fmt::format("Error reading dataset '{}': unable to get dimensions.", dataArrayPath.getTargetName())); } - auto& absDataStore = dataArray.getDataStoreRef(); - std::vector data(absDataStore.getSize()); - nonstd::span span{data.data(), data.size()}; - Result<> result; - if(start.has_value() && count.has_value()) + // Compute effective start/count for the read region + const usize rank = dims.size(); + std::vector effStart(rank, 0); + std::vector effCount(rank); + for(usize d = 0; d < rank; d++) { - result = datasetReader.readIntoSpan(span, start.value(), count.value()); + effCount[d] = dims[d]; } - else + if(start.has_value()) + { + for(usize d = 0; d < std::min(rank, start.value().size()); d++) + { + effStart[d] = start.value()[d]; + effCount[d] = dims[d] - start.value()[d]; + } + } + if(count.has_value()) { - result = datasetReader.readIntoSpan(span); + for(usize d = 0; d < std::min(rank, count.value().size()); d++) + { + effCount[d] = count.value()[d]; + } } - if(result.invalid()) + // Compute elements per row (product of all count dims except the first) + usize elementsPerRow = 1; + for(usize d = 1; d < rank; d++) { - return { - MakeErrorResult(-21003, fmt::format("Error reading dataset '{}' with '{}' total elements into data store for data array '{}' with '{}' total elements ('{}' tuples and '{}' components):\n\n{}", - dataArrayPath.getTargetName(), datasetReader.getNumElements(), dataArrayPath.toString(), dataArray.getSize(), dataArray.getNumberOfTuples(), - dataArray.getNumberOfComponents(), result.errors()[0].message))}; + elementsPerRow *= effCount[d]; + } + const usize totalRows = effCount[0]; + + // Choose batch size: read enough rows to fill ~256K elements per batch + constexpr usize k_TargetBatchElements = 262144; + const usize rowsPerBatch = std::max(static_cast(1), k_TargetBatchElements / std::max(elementsPerRow, static_cast(1))); + + const usize batchBufferSize = rowsPerBatch * elementsPerRow; + std::vector buf(batchBufferSize); + + usize flatOffset = 0; + for(usize rowStart = 0; rowStart < totalRows; rowStart += rowsPerBatch) + { + const usize rowCount = std::min(rowsPerBatch, totalRows - rowStart); + const usize batchElements = rowCount * elementsPerRow; + + // Build hyperslab for this batch within the effective region + std::vector hStart(rank); + std::vector hCount(rank); + hStart[0] = effStart[0] + rowStart; + hCount[0] = rowCount; + for(usize d = 1; d < rank; d++) + { + hStart[d] = effStart[d]; + hCount[d] = effCount[d]; + } + + nonstd::span batchSpan(buf.data(), batchElements); + auto result = datasetReader.readIntoSpan(batchSpan, hStart, hCount); + if(result.invalid()) + { + return {MakeErrorResult(-21003, fmt::format("Error reading dataset '{}' (rows {}-{}) into data store for data array '{}':\n\n{}", dataArrayPath.getTargetName(), effStart[0] + rowStart, + effStart[0] + rowStart + rowCount - 1, dataArrayPath.toString(), result.errors()[0].message))}; + } + + absDataStore.copyFromBuffer(flatOffset, nonstd::span(buf.data(), batchElements)); + flatOffset += batchElements; } - std::copy(data.begin(), data.end(), absDataStore.begin()); return {}; } diff --git a/src/simplnx/Utilities/SegmentFeatures.cpp b/src/simplnx/Utilities/SegmentFeatures.cpp index 037b7175b0..a3aaa6a212 100644 --- a/src/simplnx/Utilities/SegmentFeatures.cpp +++ b/src/simplnx/Utilities/SegmentFeatures.cpp @@ -1,9 +1,13 @@ #include "SegmentFeatures.hpp" +#include "simplnx/DataStructure/AbstractDataStore.hpp" #include "simplnx/DataStructure/Geometry/IGridGeometry.hpp" #include "simplnx/Utilities/ClusteringUtilities.hpp" #include "simplnx/Utilities/MessageHelper.hpp" +#include "simplnx/Utilities/UnionFind.hpp" +#include +#include #include using namespace nx::core; @@ -11,109 +15,154 @@ using namespace nx::core; namespace { /** - * @brief This will find the 6 face neighbor's indices. - * @param currentPoint - * @param width - * @param height - * @param depth - * @return Vector of indices + * @brief Returns the 6 face neighbor indices. When isPeriodic is true, + * boundary voxels wrap to the opposite face instead of being skipped. + * @param currentPoint Linear voxel index + * @param width X dimension + * @param height Y dimension + * @param depth Z dimension + * @param isPeriodic Whether to apply periodic boundary wrapping + * @return Vector of neighbor indices */ -std::vector getFaceNeighbors(const int64 currentPoint, const int64 width, const int64 height, const int64 depth) +std::vector getFaceNeighbors(const int64 currentPoint, const int64 width, const int64 height, const int64 depth, const bool isPeriodic) { std::vector neighbors; neighbors.reserve(6); - // decode currentPoint -> (col, row, plane) const int64 col = currentPoint % width; const int64 tmp = currentPoint / width; const int64 row = tmp % height; const int64 plane = tmp / height; - // stride for one z-slice const int64 slice = width * height; + // -X if(col > 0) { neighbors.push_back(currentPoint - 1); } + else if(isPeriodic) + { + neighbors.push_back(currentPoint + width - 1); + } + + // +X if(col < width - 1) { neighbors.push_back(currentPoint + 1); } + else if(isPeriodic) + { + neighbors.push_back(currentPoint - width + 1); + } + + // -Y if(row > 0) { neighbors.push_back(currentPoint - width); } + else if(isPeriodic) + { + neighbors.push_back(currentPoint + (height - 1) * width); + } + + // +Y if(row < height - 1) { neighbors.push_back(currentPoint + width); } + else if(isPeriodic) + { + neighbors.push_back(currentPoint - (height - 1) * width); + } + + // -Z if(plane > 0) { neighbors.push_back(currentPoint - slice); } + else if(isPeriodic) + { + neighbors.push_back(currentPoint + (depth - 1) * slice); + } + + // +Z if(plane < depth - 1) { neighbors.push_back(currentPoint + slice); } + else if(isPeriodic) + { + neighbors.push_back(currentPoint - (depth - 1) * slice); + } return neighbors; } /** - * @brief This will find all indices that are connected via the 26 face, edge or vertex neighbors - * @param currentPoint - * @param width - * @param height - * @param depth - * @return vector of indices + * @brief Returns up to 26 face/edge/vertex neighbor indices. When isPeriodic + * is true, boundary voxels wrap to the opposite face instead of being skipped. + * @param currentPoint Linear voxel index + * @param width X dimension + * @param height Y dimension + * @param depth Z dimension + * @param isPeriodic Whether to apply periodic boundary wrapping + * @return Vector of neighbor indices */ -std::vector getAllNeighbors(const int64 currentPoint, const int64 width, const int64 height, const int64 depth) +std::vector getAllNeighbors(const int64 currentPoint, const int64 width, const int64 height, const int64 depth, const bool isPeriodic) { std::vector neighbors; neighbors.reserve(26); - // decode currentPoint -> (col, row, plane) const int64 col = currentPoint % width; const int64 tmp = currentPoint / width; const int64 row = tmp % height; const int64 plane = tmp / height; - // stride for one z-slice const int64 slice = width * height; - // baseOffset == currentPoint - const int64 baseOffset = currentPoint; - for(int64 dz = -1; dz <= 1; ++dz) { - if(const int64 p = plane + dz; p < 0 || p >= depth) + int64 nz = plane + dz; + if(nz < 0 || nz >= depth) { - continue; + if(!isPeriodic) + { + continue; + } + nz = (nz + depth) % depth; } - const int64 dzOff = dz * slice; for(int64 dy = -1; dy <= 1; ++dy) { - if(const int64 r = row + dy; r < 0 || r >= height) + int64 ny = row + dy; + if(ny < 0 || ny >= height) { - continue; + if(!isPeriodic) + { + continue; + } + ny = (ny + height) % height; } - const int64 dyOff = dy * width; for(int64 dx = -1; dx <= 1; ++dx) { - // skip the center voxel itself if(dx == 0 && dy == 0 && dz == 0) { continue; } - if(int64 c = col + dx; c < 0 || c >= width) + + int64 nx = col + dx; + if(nx < 0 || nx >= width) { - continue; + if(!isPeriodic) + { + continue; + } + nx = (nx + width) % width; } - int64 neighbor = baseOffset + dzOff + dyOff + dx; - neighbors.push_back(neighbor); + + neighbors.push_back(nz * slice + ny * width + nx); } } } @@ -134,7 +183,36 @@ SegmentFeatures::SegmentFeatures(DataStructure& dataStructure, const std::atomic // ----------------------------------------------------------------------------- SegmentFeatures::~SegmentFeatures() = default; -// ----------------------------------------------------------------------------- +// ============================================================================= +// DFS Flood-Fill Segmentation (In-Core Path) +// ============================================================================= +// +// This method implements a depth-first search (DFS) flood-fill algorithm for +// segmenting voxels into features when data resides entirely in memory. +// +// Algorithm overview: +// 1. Iterate through voxels to find "seed" voxels — unassigned, valid voxels +// that start a new feature. +// 2. For each seed, assign a new feature ID (gnum) and push the seed onto a +// stack (voxelsList). +// 3. Pop voxels from the stack, examine their neighbors via the configured +// neighbor scheme (Face or FaceEdgeVertex), and call the subclass's +// determineGrouping() to decide whether a neighbor belongs to the same +// feature. If so, the neighbor is assigned the feature ID and pushed +// onto the stack for further expansion. +// 4. When the stack empties, the current feature is complete. Find the next +// seed and repeat until no seeds remain. +// +// Features are numbered in seed-discovery order (the first unassigned voxel +// encountered becomes feature 1, the next becomes feature 2, etc.). +// +// Performance note: +// This algorithm uses random-access memory patterns — the stack can pop to +// any voxel in the volume, causing non-sequential reads. This is efficient +// for in-core DataStore (O(1) random access) but extremely slow for OOC +// ZarrStore, where random access triggers chunk loads/evictions ("chunk +// thrashing"). Use executeCCL() for out-of-core datasets. +// ============================================================================= Result<> SegmentFeatures::execute(IGridGeometry* gridGeom) { ThrottledMessenger throttledMessenger = m_MessageHelper.createThrottledMessenger(); @@ -145,13 +223,20 @@ Result<> SegmentFeatures::execute(IGridGeometry* gridGeom) int64 dims[3] = {static_cast(udims[0]), static_cast(udims[1]), static_cast(udims[2])}; - // Initialize a sequence of execution modifiers + // gnum tracks the current feature ID being assigned, starting at 1. + // nextSeed is an optimization: it tracks the lowest voxel index that might + // still be unassigned, so getSeed() can skip over already-segmented voxels + // instead of rescanning from index 0 every time. int32 gnum = 1; int64 nextSeed = 0; - int64 seed = 0; // Always use the very first value of the array that we are using to segment + int64 seed = getSeed(gnum, nextSeed); + nextSeed = seed + 1; usize size = 0; - // Initialize containers + // voxelsList serves as the DFS stack (LIFO). It is pre-allocated to avoid + // frequent reallocations. 'size' is the logical stack pointer — elements + // are pushed by writing to voxelsList[size] and incrementing, and popped + // by decrementing size and reading voxelsList[size]. constexpr usize initialVoxelsListSize = 100000; std::vector voxelsList(initialVoxelsListSize, -1); @@ -163,34 +248,46 @@ Result<> SegmentFeatures::execute(IGridGeometry* gridGeom) return {}; } + // Start a new feature: push the seed onto the stack size = 0; voxelsList[size] = seed; size++; + // DFS expansion loop: pop a voxel, check its neighbors, push matches while(size > 0) { + // Pop the top of the stack (LIFO order) const int64 currentPoint = voxelsList[size - 1]; size -= 1; std::vector neighPoints; switch(m_NeighborScheme) { case NeighborScheme::Face: - neighPoints = getFaceNeighbors(currentPoint, dims[0], dims[1], dims[2]); + neighPoints = getFaceNeighbors(currentPoint, dims[0], dims[1], dims[2], m_IsPeriodic); break; case NeighborScheme::FaceEdgeVertex: - neighPoints = getAllNeighbors(currentPoint, dims[0], dims[1], dims[2]); + neighPoints = getAllNeighbors(currentPoint, dims[0], dims[1], dims[2], m_IsPeriodic); break; } for(const auto& neighbor : neighPoints) { + // determineGrouping() is implemented by the subclass. It checks whether + // the neighbor is unassigned & similar to the reference voxel, and if + // so, assigns it the current feature ID (gnum) and returns true. if(determineGrouping(currentPoint, neighbor, gnum)) { + // Push the newly-claimed neighbor onto the stack for further expansion voxelsList[size] = neighbor; size++; + // nextSeed optimization: if this neighbor was the next candidate seed, + // advance nextSeed so getSeed() won't return an already-assigned voxel. if(neighbor == nextSeed) { nextSeed = neighbor + 1; } + // If the stack has grown beyond the allocated capacity, extend it. + // The stack is stored in a flat vector, so we grow by a fixed block + // and initialize the new entries to -1. if(size >= voxelsList.size()) { size = voxelsList.size(); @@ -208,7 +305,8 @@ Result<> SegmentFeatures::execute(IGridGeometry* gridGeom) // Send a progress message float percentComplete = static_cast(totalVoxelsSegmented) / static_cast(totalVoxels) * 100.0f; throttledMessenger.sendThrottledMessage([&]() { return fmt::format("{:.2f}% - Current Feature Count: {}", percentComplete, gnum); }); - // Increment or set values for the next iteration + // Reset the stack for the next feature. assign() shrinks/grows the vector + // back to the finished feature size + 1 and fills with -1. voxelsList.assign(size + 1, -1); gnum++; // Get the next seed value @@ -216,21 +314,731 @@ Result<> SegmentFeatures::execute(IGridGeometry* gridGeom) nextSeed = seed + 1; } - m_FoundFeatures = gnum - 1; // Decrement the gnum because it will end up 1 larger than it should have been. + m_FoundFeatures = gnum - 1; // Decrement because gnum ends up 1 larger than the last assigned feature. m_MessageHelper.sendMessage(fmt::format("Total Features Found: {}", m_FoundFeatures)); return {}; } +// ============================================================================= +// Chunk-Sequential Connected Component Labeling (CCL) Algorithm +// ============================================================================= +// +// This method replaces the DFS flood-fill (execute()) with a scanline-based +// connected-component labeling algorithm optimized for out-of-core (OOC) +// data stores (e.g. ZarrStore). Unlike DFS, which accesses voxels in +// unpredictable stack-driven order, CCL processes voxels in strict Z-Y-X +// scanline order, resulting in sequential chunk access patterns that avoid +// chunk thrashing. +// +// The algorithm has three phases: +// +// Phase 1 (Forward CCL): +// Scan voxels in Z-Y-X order. For each valid voxel, examine only its +// "backward" neighbors — those already visited earlier in scanline order. +// If a backward neighbor has a label and is similar (per areNeighborsSimilar), +// adopt that label. If multiple distinct labels are found among backward +// neighbors, unite them in a Union-Find structure. If no backward neighbor +// matches, assign a fresh provisional label. Labels are written to both an +// in-memory rolling buffer (for fast neighbor lookups) and to the OOC +// featureIds store (for persistence). +// +// Phase 1b (Periodic boundary merge): +// If periodic boundaries are enabled, Phase 1 cannot detect connections +// that wrap around the volume (the wrapped neighbor has a higher linear +// index and hasn't been visited yet). This phase reads back provisional +// labels and unites similar voxels on opposite boundary faces. +// +// Phase 2 (Resolution + Relabeling): +// Flatten the Union-Find tree, then scan the featureIds store slice by +// slice in Z-Y-X order. For each provisional label, look up its Union-Find +// root and map it to a contiguous final feature ID. Write the final ID +// back in the same pass. This combined discover-and-write approach halves +// the number of OOC accesses compared to separate resolution and write +// passes, and slice-sequential iteration ensures optimal I/O. +// ============================================================================= +Result<> SegmentFeatures::executeCCL(IGridGeometry* gridGeom, AbstractDataStore& featureIdsStore) +{ + ThrottledMessenger throttledMessenger = m_MessageHelper.createThrottledMessenger(); + + const SizeVec3 udims = gridGeom->getDimensions(); + // getDimensions() returns [X, Y, Z] + const int64 dimX = static_cast(udims[0]); + const int64 dimY = static_cast(udims[1]); + const int64 dimZ = static_cast(udims[2]); + const usize totalVoxels = static_cast(dimX) * static_cast(dimY) * static_cast(dimZ); + + const int64 sliceStride = dimX * dimY; + + const bool useFaceOnly = (m_NeighborScheme == NeighborScheme::Face); + + UnionFind unionFind; + int32 nextLabel = 1; // Provisional labels start at 1 + + // Rolling 2-slice buffer for backward neighbor label lookups. + // + // Why 2 slices is sufficient: + // In Z-Y-X scanline order, a voxel at (ix, iy, iz) has backward neighbors + // only in the current Z-slice (iz) or the immediately previous Z-slice + // (iz-1). No backward neighbor can ever be in Z-slice (iz-2) or earlier, + // because all 13 backward neighbor offsets have dz in {-1, 0}. Therefore, + // keeping just 2 slices in memory — the current and the previous — is + // enough for all backward neighbor label reads. + // + // This design uses O(dimX * dimY) memory instead of O(dimX * dimY * dimZ), + // enabling processing of datasets much larger than available RAM. + // + // Buffer layout: Z-slice (iz % 2) occupies indices + // [sliceOffset .. sliceOffset + sliceStride), where + // sliceOffset = (iz % 2) * sliceSize. + const usize sliceSize = static_cast(sliceStride); + std::vector labelBuffer(2 * sliceSize, 0); + + // ========================================================================= + // Phase 1: Forward CCL - assign provisional labels using backward neighbors + // ========================================================================= + m_MessageHelper.sendMessage("Forward CCL pass..."); + + // Slice buffer for batch-writing featureIds to the data store. + // Phase 1 only writes to featureIdsStore (never reads), so we accumulate + // writes per Z-slice and flush once via copyFromBuffer at slice end. + std::vector featureIdsSlice(sliceSize, 0); + + for(int64 iz = 0; iz < dimZ; iz++) + { + if(m_ShouldCancel) + { + return {}; + } + + prepareForSlice(iz, dimX, dimY, dimZ); + + // Zero the featureIds slice buffer for this Z-slice (prevents stale + // labels from the previous slice being written back for invalid voxels) + std::fill(featureIdsSlice.begin(), featureIdsSlice.end(), 0); + + // Clear the current slice's portion of the rolling buffer + const usize currentSliceOffset = static_cast(iz % 2) * sliceSize; + std::fill(labelBuffer.begin() + currentSliceOffset, labelBuffer.begin() + currentSliceOffset + sliceSize, 0); + + // Clear the featureIds slice buffer for this Z-slice + std::fill(featureIdsSlice.begin(), featureIdsSlice.end(), 0); + + for(int64 iy = 0; iy < dimY; iy++) + { + for(int64 ix = 0; ix < dimX; ix++) + { + const int64 index = iz * sliceStride + iy * dimX + ix; + const usize bufIdx = currentSliceOffset + static_cast(iy * dimX + ix); + + // Skip voxels that are not valid + if(!isValidVoxel(index)) + { + continue; + } + + // Check backward neighbors for existing labels. + // "Backward" neighbors are those with a smaller linear index — i.e., + // already processed earlier in Z-Y-X scanline order. In 3D, these are + // neighbors with dz < 0, or dz == 0 && dy < 0, or dz == 0 && dy == 0 + // && dx < 0. Forward neighbors (higher linear index) are not yet + // labeled and cannot be consulted. + // + // Neighbor labels are read from the rolling buffer (direct memory + // access, O(1)) rather than from the OOC featureIds store, avoiding + // chunk loads for every neighbor lookup. + int32 assignedLabel = 0; + const usize prevSliceOffset = static_cast((iz + 1) % 2) * sliceSize; + + if(useFaceOnly) + { + // Face connectivity: exactly 3 backward neighbors exist: + // -X (dx=-1): one column to the left in the same row/slice + // -Y (dy=-1): one row earlier in the same slice + // -Z (dz=-1): same (x,y) position in the previous slice + // The 3 forward neighbors (+X, +Y, +Z) have not been labeled yet + // and are skipped. + + // Check -X neighbor (same Z-slice, same buffer region) + if(ix > 0) + { + const int64 neighIdx = index - 1; + int32 neighLabel = labelBuffer[bufIdx - 1]; + if(neighLabel > 0 && areNeighborsSimilar(index, neighIdx)) + { + if(assignedLabel == 0) + { + assignedLabel = neighLabel; + } + else if(assignedLabel != neighLabel) + { + unionFind.unite(assignedLabel, neighLabel); + } + } + } + // Check -Y neighbor (same Z-slice, same buffer region) + if(iy > 0) + { + const int64 neighIdx = index - dimX; + int32 neighLabel = labelBuffer[currentSliceOffset + static_cast((iy - 1) * dimX + ix)]; + if(neighLabel > 0 && areNeighborsSimilar(index, neighIdx)) + { + if(assignedLabel == 0) + { + assignedLabel = neighLabel; + } + else if(assignedLabel != neighLabel) + { + unionFind.unite(assignedLabel, neighLabel); + } + } + } + // Check -Z neighbor (previous Z-slice, other buffer region) + if(iz > 0) + { + const int64 neighIdx = index - sliceStride; + int32 neighLabel = labelBuffer[prevSliceOffset + static_cast(iy * dimX + ix)]; + if(neighLabel > 0 && areNeighborsSimilar(index, neighIdx)) + { + if(assignedLabel == 0) + { + assignedLabel = neighLabel; + } + else if(assignedLabel != neighLabel) + { + unionFind.unite(assignedLabel, neighLabel); + } + } + } + } + else + { + // FaceEdgeVertex connectivity: 13 backward neighbors out of 26 total. + // + // A 3x3x3 neighborhood has 26 neighbors (excluding self). Exactly + // half (13) have a smaller linear index in Z-Y-X order and are thus + // "backward." These are enumerated by iterating: + // dz in {-1, 0}: + // dz=-1: all 9 neighbors in the previous Z-slice (any dx, dy) + // dz= 0: only neighbors with dy < 0 (3 neighbors), or + // dy == 0 && dx == -1 (1 neighbor) => 4 total + // Total: 9 + 4 = 13 backward neighbors + // + // The loop bounds below encode this enumeration efficiently: + // - dz ranges [-1, 0] + // - dy ranges [-1, +1] when dz<0, or [-1, 0] when dz==0 + // - dx ranges [-1, +1] when dz<0 or dy<0, or [-1, -1] when dz==0 && dy==0 + for(int64 dz = -1; dz <= 0; ++dz) + { + const int64 nz = iz + dz; + if(nz < 0 || nz >= dimZ) + { + continue; + } + + const usize neighSliceOffset = (dz < 0) ? prevSliceOffset : currentSliceOffset; + + const int64 dyStart = -1; + const int64 dyEnd = (dz < 0) ? 1 : 0; + + for(int64 dy = dyStart; dy <= dyEnd; ++dy) + { + const int64 ny = iy + dy; + if(ny < 0 || ny >= dimY) + { + continue; + } + + int64 dxStart; + int64 dxEnd; + if(dz < 0) + { + dxStart = -1; + dxEnd = 1; + } + else if(dy < 0) + { + dxStart = -1; + dxEnd = 1; + } + else + { + dxStart = -1; + dxEnd = -1; + } + + for(int64 dx = dxStart; dx <= dxEnd; ++dx) + { + const int64 nx = ix + dx; + if(nx < 0 || nx >= dimX) + { + continue; + } + if(dx == 0 && dy == 0 && dz == 0) + { + continue; + } + + const int64 neighIdx = nz * sliceStride + ny * dimX + nx; + int32 neighLabel = labelBuffer[neighSliceOffset + static_cast(ny * dimX + nx)]; + if(neighLabel > 0 && areNeighborsSimilar(index, neighIdx)) + { + if(assignedLabel == 0) + { + assignedLabel = neighLabel; + } + else if(assignedLabel != neighLabel) + { + unionFind.unite(assignedLabel, neighLabel); + } + } + } + } + } + } + + // If no matching backward neighbor, assign new provisional label + if(assignedLabel == 0) + { + assignedLabel = nextLabel++; + unionFind.find(assignedLabel); // Initialize in union-find + } + + // Write label to rolling buffer (for neighbor reads) and slice buffer (for batch write) + labelBuffer[bufIdx] = assignedLabel; + const usize inSlice = static_cast(iy * dimX + ix); + featureIdsSlice[inSlice] = assignedLabel; + } + } + + // Batch-write this Z-slice's featureIds to the data store + featureIdsStore.copyFromBuffer(static_cast(iz) * sliceSize, nonstd::span(featureIdsSlice.data(), sliceSize)); + + // Send progress per Z-slice + float percentComplete = static_cast(iz + 1) / static_cast(dimZ) * 100.0f; + throttledMessenger.sendThrottledMessage([percentComplete]() { return fmt::format("Forward CCL: {:.1f}% complete", percentComplete); }); + } + + if(m_ShouldCancel) + { + return {}; + } + + // ========================================================================= + // Phase 1b: Periodic boundary merge (O(slice) memory) + // ========================================================================= + // The forward CCL pass cannot detect connections that wrap around periodic + // boundaries because the wrapped neighbor has a higher linear index and + // has not been processed yet when the boundary voxel is visited. This + // phase reads back provisional labels from featureIdsStore slice by slice + // and unites labels of similar voxels on opposite boundary faces. + // + // Memory strategy: instead of reading the entire volume into memory, we + // read featureIds one or two Z-slices at a time and call prepareForSlice() + // so areNeighborsSimilar() reads from the subclass's fast slice buffers. + // This keeps memory at O(dimX * dimY) rather than O(dimX * dimY * dimZ). + if(m_IsPeriodic) + { + m_MessageHelper.sendMessage("Merging periodic boundaries..."); + + // Reusable slice-sized buffers for reading featureIds from the store + std::vector featureIdsSliceCur(sliceSize, 0); + + if(useFaceOnly) + { + // Face connectivity: each axis is handled independently because face + // neighbors only connect along a single axis. For each axis, we + // iterate over the 2D face and compare each voxel at the low boundary + // (e.g. ix=0) with its counterpart at the high boundary (e.g. + // ix=dimX-1). These are the same voxel pairs that getFaceNeighbors() + // would return with isPeriodic=true, but which Phase 1 could not + // process because the wrapped neighbor had not yet been labeled. + + // X-axis: unite voxels at ix=0 with ix=dimX-1 + // Both voxels are in the same Z-slice, so one slice suffices. + if(dimX > 1) + { + for(int64 iz = 0; iz < dimZ; iz++) + { + featureIdsStore.copyIntoBuffer(static_cast(iz) * sliceSize, nonstd::span(featureIdsSliceCur.data(), sliceSize)); + prepareForSlice(iz, dimX, dimY, dimZ); + + for(int64 iy = 0; iy < dimY; iy++) + { + const int64 idxA = iz * sliceStride + iy * dimX; + const int64 idxB = iz * sliceStride + iy * dimX + (dimX - 1); + const int32 labelA = featureIdsSliceCur[static_cast(iy * dimX)]; + const int32 labelB = featureIdsSliceCur[static_cast(iy * dimX + (dimX - 1))]; + if(labelA > 0 && labelB > 0 && areNeighborsSimilar(idxA, idxB)) + { + unionFind.unite(labelA, labelB); + } + } + } + } + + // Y-axis: unite voxels at iy=0 with iy=dimY-1 + // Both voxels are in the same Z-slice, so one slice suffices. + if(dimY > 1) + { + for(int64 iz = 0; iz < dimZ; iz++) + { + featureIdsStore.copyIntoBuffer(static_cast(iz) * sliceSize, nonstd::span(featureIdsSliceCur.data(), sliceSize)); + prepareForSlice(iz, dimX, dimY, dimZ); + + for(int64 ix = 0; ix < dimX; ix++) + { + const int64 idxA = iz * sliceStride + ix; + const int64 idxB = iz * sliceStride + (dimY - 1) * dimX + ix; + const int32 labelA = featureIdsSliceCur[static_cast(ix)]; + const int32 labelB = featureIdsSliceCur[static_cast((dimY - 1) * dimX + ix)]; + if(labelA > 0 && labelB > 0 && areNeighborsSimilar(idxA, idxB)) + { + unionFind.unite(labelA, labelB); + } + } + } + } + + // Z-axis: unite voxels at iz=0 with iz=dimZ-1 + // These are in different Z-slices, so we read both into separate + // buffers. We call prepareForSlice for both so the 2-slot rolling + // buffer holds both slices for areNeighborsSimilar(). + if(dimZ > 1) + { + std::vector featureIdsSliceOther(sliceSize, 0); + featureIdsStore.copyIntoBuffer(0, nonstd::span(featureIdsSliceCur.data(), sliceSize)); + featureIdsStore.copyIntoBuffer(static_cast(dimZ - 1) * sliceSize, nonstd::span(featureIdsSliceOther.data(), sliceSize)); + + // Load both slices into the subclass's 2-slot rolling buffer so + // areNeighborsSimilar() can compare voxels across these two slices. + prepareForSlice(0, dimX, dimY, dimZ); + prepareForSlice(dimZ - 1, dimX, dimY, dimZ); + + for(int64 iy = 0; iy < dimY; iy++) + { + for(int64 ix = 0; ix < dimX; ix++) + { + const usize inSlice = static_cast(iy * dimX + ix); + const int64 idxA = iy * dimX + ix; + const int64 idxB = (dimZ - 1) * sliceStride + iy * dimX + ix; + const int32 labelA = featureIdsSliceCur[inSlice]; + const int32 labelB = featureIdsSliceOther[inSlice]; + if(labelA > 0 && labelB > 0 && areNeighborsSimilar(idxA, idxB)) + { + unionFind.unite(labelA, labelB); + } + } + } + } + } + else + { + // FaceEdgeVertex connectivity: check all 26-neighbor pairs that wrap + // across periodic boundaries. Unlike face-only mode, edge and vertex + // neighbors can wrap across two or even three axes simultaneously + // (e.g. a corner voxel's diagonal neighbor wraps in X, Y, and Z). + // + // Memory strategy: iterate Z-slices one at a time. For each slice, + // read its featureIds into featureIdsSliceCur. Only Z-boundary slices + // (iz=0 and iz=dimZ-1) can have neighbors that wrap in Z; all other + // slices only have X/Y wrapping where both voxels are in the same + // Z-slice. For Z-boundary slices, we also read the wrapped partner + // slice (dimZ-1 or 0) into featureIdsSliceWrapped. + // + // The subclass's prepareForSlice 2-slot rolling buffer naturally + // holds both the current slice and the wrapped partner, so + // areNeighborsSimilar() works correctly. + // + // Pre-read featureIds for slices 0 and dimZ-1 into persistent buffers + // so they are available when either Z-boundary slice is processed. + std::vector featureIdsSlice0(sliceSize, 0); + std::vector featureIdsSliceLast(sliceSize, 0); + featureIdsStore.copyIntoBuffer(0, nonstd::span(featureIdsSlice0.data(), sliceSize)); + if(dimZ > 1) + { + featureIdsStore.copyIntoBuffer(static_cast(dimZ - 1) * sliceSize, nonstd::span(featureIdsSliceLast.data(), sliceSize)); + } + + for(int64 iz = 0; iz < dimZ; iz++) + { + if(m_ShouldCancel) + { + return {}; + } + + // Use the pre-read buffer for slices 0 and dimZ-1; read fresh for others + if(iz == 0) + { + std::copy(featureIdsSlice0.begin(), featureIdsSlice0.end(), featureIdsSliceCur.begin()); + } + else if(iz == dimZ - 1) + { + std::copy(featureIdsSliceLast.begin(), featureIdsSliceLast.end(), featureIdsSliceCur.begin()); + } + else + { + featureIdsStore.copyIntoBuffer(static_cast(iz) * sliceSize, nonstd::span(featureIdsSliceCur.data(), sliceSize)); + } + + // Load input data for the current slice + prepareForSlice(iz, dimX, dimY, dimZ); + + // Determine if this is a Z-boundary slice and identify the wrapped + // partner. Only iz=0 can wrap to dimZ-1, and only iz=dimZ-1 can + // wrap to 0. Interior slices have no Z-wrapped neighbors. + int64 wrappedPartnerZ = -1; // sentinel: no Z-wrapped partner + const int32* wrappedSlicePtr = nullptr; + if(iz == 0 && dimZ > 1) + { + wrappedPartnerZ = dimZ - 1; + wrappedSlicePtr = featureIdsSliceLast.data(); + // Load the wrapped partner into the 2-slot buffer so + // areNeighborsSimilar() can access both slices + prepareForSlice(wrappedPartnerZ, dimX, dimY, dimZ); + } + else if(iz == dimZ - 1 && dimZ > 1) + { + wrappedPartnerZ = 0; + wrappedSlicePtr = featureIdsSlice0.data(); + prepareForSlice(wrappedPartnerZ, dimX, dimY, dimZ); + } + + for(int64 iy = 0; iy < dimY; iy++) + { + for(int64 ix = 0; ix < dimX; ix++) + { + // Only boundary voxels can have neighbors that wrap around + const bool onBoundary = (ix == 0 || ix == dimX - 1 || iy == 0 || iy == dimY - 1 || iz == 0 || iz == dimZ - 1); + if(!onBoundary) + { + continue; + } + + const usize inSlice = static_cast(iy * dimX + ix); + const int64 index = iz * sliceStride + iy * dimX + ix; + const int32 labelCurrent = featureIdsSliceCur[inSlice]; + if(labelCurrent <= 0) + { + continue; + } + + for(int64 dz = -1; dz <= 1; ++dz) + { + int64 nz = iz + dz; + bool wrappedZ = false; + if(nz < 0) + { + nz += dimZ; + wrappedZ = true; + } + else if(nz >= dimZ) + { + nz -= dimZ; + wrappedZ = true; + } + + for(int64 dy = -1; dy <= 1; ++dy) + { + int64 ny = iy + dy; + bool wrappedY = false; + if(ny < 0) + { + ny += dimY; + wrappedY = true; + } + else if(ny >= dimY) + { + ny -= dimY; + wrappedY = true; + } + + for(int64 dx = -1; dx <= 1; ++dx) + { + if(dx == 0 && dy == 0 && dz == 0) + { + continue; + } + + int64 nx = ix + dx; + bool wrappedX = false; + if(nx < 0) + { + nx += dimX; + wrappedX = true; + } + else if(nx >= dimX) + { + nx -= dimX; + wrappedX = true; + } + + // Only process pairs that actually wrap around at least one + // axis. Non-wrapped pairs were already handled in Phase 1. + if(!wrappedX && !wrappedY && !wrappedZ) + { + continue; + } + + const int64 neighIdx = nz * sliceStride + ny * dimX + nx; + // Deduplication: only process the pair where neighIdx > index. + // This ensures each (voxelA, voxelB) pair is united exactly + // once, since unite() is symmetric. + if(neighIdx <= index) + { + continue; + } + + // Look up the neighbor's label from the appropriate slice buffer. + // If the neighbor is in the same Z-slice, use featureIdsSliceCur. + // If the neighbor is in the wrapped partner Z-slice, use wrappedSlicePtr. + int32 labelNeigh = 0; + if(nz == iz) + { + labelNeigh = featureIdsSliceCur[static_cast(ny * dimX + nx)]; + } + else if(nz == wrappedPartnerZ && wrappedSlicePtr != nullptr) + { + labelNeigh = wrappedSlicePtr[static_cast(ny * dimX + nx)]; + } + + if(labelNeigh > 0 && areNeighborsSimilar(index, neighIdx)) + { + unionFind.unite(labelCurrent, labelNeigh); + } + } + } + } + } + } + } + } + } + + if(m_ShouldCancel) + { + return {}; + } + + // ========================================================================= + // Phase 2: Resolution + Relabeling (combined single pass) + // ========================================================================= + // + // After Phase 1/1b, every valid voxel has a provisional label and the + // Union-Find knows which provisional labels belong to the same connected + // component. This phase: + // 1. Flattens the Union-Find so every label points directly to its root + // (path compression eliminates intermediate nodes). + // 2. Scans voxels slice-by-slice in Z-Y-X order. For each + // provisional label, performs a two-level lookup: + // a) label -> root: via unionFind.find(label) (O(1) after flatten) + // b) root -> finalId: via the labelToFinal[] map + // If the root has not yet been assigned a final ID, allocate the next + // sequential ID (finalFeatureCount++). Then cache the mapping for the + // original label as well (labelToFinal[label] = finalId) so subsequent + // voxels with the same provisional label skip the union-find lookup. + // 3. Writes the final ID back to featureIdsStore via copyFromBuffer. + // + // Combining discovery and relabeling into a single pass halves the number + // of OOC accesses compared to doing them separately. The slice-sequential + // iteration order ensures optimal I/O patterns. + // + // Because the scan is in linear (Z-Y-X) order, final feature IDs are + // assigned in the order their first voxel appears in the volume, matching + // the seed-discovery order of the DFS algorithm. + // ========================================================================= + m_MessageHelper.sendMessage("Resolving labels and writing final feature IDs..."); + + unionFind.flatten(); + + // labelToFinal maps provisional label -> final contiguous feature ID. + // Indexed by provisional label (0..nextLabel-1). A value of 0 means + // "not yet assigned." This avoids a hash map and gives O(1) lookups. + std::vector labelToFinal(static_cast(nextLabel), 0); + int32 finalFeatureCount = 0; + + std::vector sliceData(sliceSize); + + for(int64 iz = 0; iz < dimZ; iz++) + { + if(m_ShouldCancel) + { + return {}; + } + + featureIdsStore.copyIntoBuffer(static_cast(iz) * sliceSize, nonstd::span(sliceData.data(), sliceSize)); + + for(int64 iy = 0; iy < dimY; iy++) + { + for(int64 ix = 0; ix < dimX; ix++) + { + const usize inSlice = static_cast(iy * dimX + ix); + int32 label = sliceData[inSlice]; + if(label > 0) + { + // Two-level lookup: provisional label -> union-find root -> final ID + if(labelToFinal[label] == 0) + { + // Level 1: find this label's root in the (flattened) union-find + int32 root = static_cast(unionFind.find(label)); + // Level 2: if the root hasn't been assigned a final ID yet, + // allocate the next sequential feature ID + if(labelToFinal[root] == 0) + { + finalFeatureCount++; + labelToFinal[root] = finalFeatureCount; + } + // Cache the mapping for this provisional label so future voxels + // with the same label skip the union-find lookup entirely + labelToFinal[label] = labelToFinal[root]; + } + // Write the final contiguous feature ID back to the slice buffer + sliceData[inSlice] = labelToFinal[label]; + } + } + } + + featureIdsStore.copyFromBuffer(static_cast(iz) * sliceSize, nonstd::span(sliceData.data(), sliceSize)); + + // Send progress + float percentComplete = static_cast(iz + 1) / static_cast(dimZ) * 100.0f; + throttledMessenger.sendThrottledMessage([percentComplete]() { return fmt::format("Relabeling: {:.1f}% complete", percentComplete); }); + } + + m_FoundFeatures = finalFeatureCount; + m_MessageHelper.sendMessage(fmt::format("Total Features Found: {}", m_FoundFeatures)); + + return {}; +} + +// ----------------------------------------------------------------------------- int64 SegmentFeatures::getSeed(int32 gnum, int64 nextSeed) const { return -1; } +// ----------------------------------------------------------------------------- bool SegmentFeatures::determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const { return false; } +// ----------------------------------------------------------------------------- +void SegmentFeatures::prepareForSlice(int64 /*iz*/, int64 /*dimX*/, int64 /*dimY*/, int64 /*dimZ*/) +{ +} + +// ----------------------------------------------------------------------------- +bool SegmentFeatures::isValidVoxel(int64 point) const +{ + return true; +} + +// ----------------------------------------------------------------------------- +bool SegmentFeatures::areNeighborsSimilar(int64 point1, int64 point2) const +{ + return false; +} + // ----------------------------------------------------------------------------- SegmentFeatures::SeedGenerator SegmentFeatures::initializeStaticVoxelSeedGenerator() const { diff --git a/src/simplnx/Utilities/SegmentFeatures.hpp b/src/simplnx/Utilities/SegmentFeatures.hpp index 1c015485e2..aa1a65e8b3 100644 --- a/src/simplnx/Utilities/SegmentFeatures.hpp +++ b/src/simplnx/Utilities/SegmentFeatures.hpp @@ -17,6 +17,8 @@ namespace nx::core { class IGridGeometry; +template +class AbstractDataStore; namespace segment_features { @@ -51,16 +53,25 @@ class SIMPLNX_EXPORT SegmentFeatures }; /** - * @brief execute + * @brief Original DFS-based segmentation (in-core optimized). * @param gridGeom * @return */ Result<> execute(IGridGeometry* gridGeom); + /** + * @brief Chunk-sequential CCL-based segmentation optimized for out-of-core. + * + * Subclasses must override isValidVoxel() and areNeighborsSimilar() to use this code path. + * + * @param gridGeom The grid geometry providing dimensions and neighbor offsets. + * @param featureIdsStore The data store to write assigned feature IDs into. + * @return Result indicating success or an error with a descriptive message. + */ + Result<> executeCCL(IGridGeometry* gridGeom, AbstractDataStore& featureIdsStore); + /** * @brief Returns the seed for the specified values. - * @param data - * @param args * @param gnum * @param nextSeed * @return int64 @@ -69,8 +80,6 @@ class SIMPLNX_EXPORT SegmentFeatures /** * @brief Determines the grouping for the specified values. - * @param data - * @param args * @param referencePoint * @param neighborPoint * @param gnum @@ -82,7 +91,6 @@ class SIMPLNX_EXPORT SegmentFeatures * @brief * @param featureIds * @param totalFeatures - * @param distribution */ void randomizeFeatureIds(Int32Array* featureIds, uint64 totalFeatures); @@ -106,8 +114,53 @@ class SIMPLNX_EXPORT SegmentFeatures { return false; } + + /** + * @brief Pure data comparison without featureId assignment. + * Used by the CCL algorithm which handles label assignment separately. + * @param index First voxel index + * @param neighIndex Second voxel index + * @return true if the two voxels should be in the same feature + */ + virtual bool compare(int64 index, int64 neighIndex) + { + return false; + } }; + /** + * @brief Can this voxel be a feature member? (mask + phase check, NO featureId check) + * Default returns true (all voxels are valid). + * @param point Linear voxel index + * @return true if this voxel can participate in segmentation + */ + virtual bool isValidVoxel(int64 point) const; + + /** + * @brief Should these two adjacent voxels be in the same feature? (data comparison only) + * Default returns false (no voxels are similar). + * @param point1 First voxel index + * @param point2 Second voxel index + * @return true if the two voxels should be grouped together + */ + virtual bool areNeighborsSimilar(int64 point1, int64 point2) const; + + /** + * @brief Called by executeCCL at the start of each Z-slice to allow subclasses + * to pre-load input data into local buffers, eliminating per-element OOC overhead + * during neighbor comparisons. + * + * Called with iz = -1 before Phase 1b (periodic boundary merge) to signal that + * buffering should be disabled, since Phase 1b may access arbitrary Z-slices. + * + * Default implementation does nothing. + * @param iz Current Z-slice index, or -1 to disable buffering. + * @param dimX X dimension of the grid. + * @param dimY Y dimension of the grid. + * @param dimZ Z dimension of the grid. + */ + virtual void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ); + protected: DataStructure& m_DataStructure; bool m_IsPeriodic = false; diff --git a/src/simplnx/Utilities/SliceBufferedTransfer.hpp b/src/simplnx/Utilities/SliceBufferedTransfer.hpp new file mode 100644 index 0000000000..737e5e2208 --- /dev/null +++ b/src/simplnx/Utilities/SliceBufferedTransfer.hpp @@ -0,0 +1,235 @@ +#pragma once + +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/Utilities/FilterUtilities.hpp" + +#include + +#include +#include +#include +#include +#include + +namespace nx::core +{ + +/** + * @brief Performs Z-slice buffered tuple transfer for a single typed DataArray. + * + * Reads source and destination Z-slices into memory, copies tuples in-memory + * based on a neighbors mapping, then writes back. This eliminates per-element + * OOC store access during transfer phases. + * + * Requires that source indices are always within ±1 Z-slice of the destination + * (true for face-neighbor morphological algorithms). + */ +struct SliceBufferedTransferFunctor +{ + template + using BufferType = std::conditional_t, std::unique_ptr, std::vector>; + + template + static T* bufPtr(std::vector& v) + { + return v.data(); + } + + template + static T* bufPtr(std::unique_ptr& p) + { + return p.get(); + } + + template + static const T* bufPtr(const std::vector& v) + { + return v.data(); + } + + template + static const T* bufPtr(const std::unique_ptr& p) + { + return p.get(); + } + + template + static BufferType makeBuf(usize size) + { + if constexpr(std::is_same_v) + { + return std::make_unique(size); + } + else + { + return std::vector(size); + } + } + + template + void operator()(IDataArray& dataArray, const std::vector& neighbors, usize sliceSize, usize dimZ, const std::function& shouldCopy) + { + auto& store = dynamic_cast&>(dataArray).getDataStoreRef(); + const usize numComp = store.getNumberOfComponents(); + const usize sliceValues = sliceSize * numComp; + + // Rolling source window: slot 0=z-1, slot 1=z, slot 2=z+1 + std::array, 3> srcSlices; + for(auto& s : srcSlices) + { + s = makeBuf(sliceValues); + } + auto destSlice = makeBuf(sliceValues); + + auto readSlice = [&](usize z, usize slot) { store.copyIntoBuffer(z * sliceValues, nonstd::span(bufPtr(srcSlices[slot]), sliceValues)); }; + + // Initialize source rolling window + readSlice(0, 1); + if(dimZ > 1) + { + readSlice(1, 2); + } + + for(usize zIdx = 0; zIdx < dimZ; zIdx++) + { + if(zIdx > 0) + { + std::swap(srcSlices[0], srcSlices[1]); + std::swap(srcSlices[1], srcSlices[2]); + if(zIdx + 1 < dimZ) + { + readSlice(zIdx + 1, 2); + } + } + + // Read dest slice + store.copyIntoBuffer(zIdx * sliceValues, nonstd::span(bufPtr(destSlice), sliceValues)); + + bool modified = false; + for(usize inSlice = 0; inSlice < sliceSize; inSlice++) + { + const usize destIdx = zIdx * sliceSize + inSlice; + const int64 srcIdx = neighbors[destIdx]; + if(srcIdx >= 0 && shouldCopy(destIdx)) + { + const usize srcZ = static_cast(srcIdx) / sliceSize; + const usize srcInSlice = static_cast(srcIdx) % sliceSize; + usize srcSlot = 1; + if(srcZ < zIdx) + { + srcSlot = 0; + } + else if(srcZ > zIdx) + { + srcSlot = 2; + } + + for(usize c = 0; c < numComp; c++) + { + bufPtr(destSlice)[inSlice * numComp + c] = bufPtr(srcSlices[srcSlot])[srcInSlice * numComp + c]; + } + modified = true; + } + } + + if(modified) + { + store.copyFromBuffer(zIdx * sliceValues, nonstd::span(bufPtr(destSlice), sliceValues)); + } + } + } +}; + +/** + * @brief Convenience function to perform slice-buffered transfer on a single IDataArray. + * Dispatches on the array's DataType to call the typed implementation. + */ +inline void SliceBufferedTransfer(IDataArray& dataArray, const std::vector& neighbors, usize sliceSize, usize dimZ, const std::function& shouldCopy) +{ + ExecuteDataFunction(SliceBufferedTransferFunctor{}, dataArray.getDataType(), dataArray, neighbors, sliceSize, dimZ, shouldCopy); +} + +/** + * @brief Transfers a single Z-slice of a typed DataArray using per-slice marks. + * + * For each voxel in the dest Z-slice where marks[inSlice] >= 0, copies the tuple + * from the source location (which must be within ±1 Z of destZ). Manages its own + * 3-slice source rolling window internally. + * + * This is the building block for O(sliceSize) memory algorithms that eliminate + * full-volume neighbor arrays. + */ +struct SliceTransferOneZFunctor +{ + template + void operator()(IDataArray& dataArray, const std::vector& sliceMarks, usize sliceSize, usize destZ, usize dimZ) + { + using BufT = typename SliceBufferedTransferFunctor::BufferType; + auto& store = dynamic_cast&>(dataArray).getDataStoreRef(); + const usize numComp = store.getNumberOfComponents(); + const usize sliceValues = sliceSize * numComp; + + // Read dest slice + auto destBuf = SliceBufferedTransferFunctor::makeBuf(sliceValues); + store.copyIntoBuffer(destZ * sliceValues, nonstd::span(SliceBufferedTransferFunctor::bufPtr(destBuf), sliceValues)); + + // Read source slices on demand (only those needed) + std::array srcBufs; // 0=z-1, 1=z, 2=z+1 + std::array srcLoaded = {false, false, false}; + + auto ensureSrcLoaded = [&](usize slot, usize srcZ) { + if(!srcLoaded[slot] && srcZ < dimZ) + { + srcBufs[slot] = SliceBufferedTransferFunctor::makeBuf(sliceValues); + store.copyIntoBuffer(srcZ * sliceValues, nonstd::span(SliceBufferedTransferFunctor::bufPtr(srcBufs[slot]), sliceValues)); + srcLoaded[slot] = true; + } + }; + + bool modified = false; + for(usize inSlice = 0; inSlice < sliceSize; inSlice++) + { + const int64 srcGlobalIdx = sliceMarks[inSlice]; + if(srcGlobalIdx < 0) + { + continue; + } + + const usize srcZ = static_cast(srcGlobalIdx) / sliceSize; + const usize srcInSlice = static_cast(srcGlobalIdx) % sliceSize; + + usize srcSlot = 1; + if(srcZ < destZ) + { + srcSlot = 0; + } + else if(srcZ > destZ) + { + srcSlot = 2; + } + + ensureSrcLoaded(srcSlot, srcZ); + + for(usize c = 0; c < numComp; c++) + { + SliceBufferedTransferFunctor::bufPtr(destBuf)[inSlice * numComp + c] = SliceBufferedTransferFunctor::bufPtr(srcBufs[srcSlot])[srcInSlice * numComp + c]; + } + modified = true; + } + + if(modified) + { + store.copyFromBuffer(destZ * sliceValues, nonstd::span(SliceBufferedTransferFunctor::bufPtr(destBuf), sliceValues)); + } + } +}; + +/** + * @brief Convenience function to transfer a single Z-slice using per-slice marks. + */ +inline void SliceBufferedTransferOneZ(IDataArray& dataArray, const std::vector& sliceMarks, usize sliceSize, usize destZ, usize dimZ) +{ + ExecuteDataFunction(SliceTransferOneZFunctor{}, dataArray.getDataType(), dataArray, sliceMarks, sliceSize, destZ, dimZ); +} + +} // namespace nx::core diff --git a/src/simplnx/Utilities/UnionFind.hpp b/src/simplnx/Utilities/UnionFind.hpp new file mode 100644 index 0000000000..6fe6247748 --- /dev/null +++ b/src/simplnx/Utilities/UnionFind.hpp @@ -0,0 +1,188 @@ +#pragma once + +#include "simplnx/simplnx_export.hpp" + +#include "simplnx/Common/Types.hpp" + +#include +#include + +namespace nx::core +{ + +/** + * @class UnionFind + * @brief Vector-based Union-Find (Disjoint Set) data structure for tracking + * connected component equivalences during chunk-sequential processing. + * + * Uses union-by-rank and path-halving compression for near-O(1) amortized + * find() and unite() operations. Internal storage uses contiguous vectors + * indexed by label for cache-friendly access (no hash map overhead). + * + * Key features: + * - Labels are contiguous integers starting from 1 (0 is unused/invalid) + * - Grows dynamically as new labels are encountered + * - Path halving in find() for near-O(1) amortized lookups + * - Union-by-rank for balanced merges + * - Accumulates sizes at each label during construction + * - Single-pass flatten() for full path compression and size accumulation + */ +class SIMPLNX_EXPORT UnionFind +{ +public: + UnionFind() + { + // Index 0 is unused (labels start at 1). Initialize with a small capacity. + constexpr usize k_InitialCapacity = 64; + m_Parent.resize(k_InitialCapacity); + m_Rank.resize(k_InitialCapacity, 0); + m_Size.resize(k_InitialCapacity, 0); + // Initialize all entries as self-parents + for(usize i = 0; i < k_InitialCapacity; i++) + { + m_Parent[i] = static_cast(i); + } + } + + ~UnionFind() = default; + + UnionFind(const UnionFind&) = delete; + UnionFind(UnionFind&&) noexcept = default; + UnionFind& operator=(const UnionFind&) = delete; + UnionFind& operator=(UnionFind&&) noexcept = default; + + /** + * @brief Find the root label with path-halving compression. + * Each node on the path is redirected to its grandparent, giving + * near-O(1) amortized performance. + * @param x Label to find + * @return Root label + */ + int64 find(int64 x) + { + ensureCapacity(x); + + // Path halving: point each node to its grandparent while walking + while(m_Parent[x] != x) + { + m_Parent[x] = m_Parent[m_Parent[x]]; + x = m_Parent[x]; + } + return x; + } + + /** + * @brief Unite two labels into the same equivalence class using union-by-rank. + * @param a First label + * @param b Second label + */ + void unite(int64 a, int64 b) + { + int64 rootA = find(a); + int64 rootB = find(b); + + if(rootA == rootB) + { + return; + } + + if(m_Rank[rootA] < m_Rank[rootB]) + { + m_Parent[rootA] = rootB; + } + else if(m_Rank[rootA] > m_Rank[rootB]) + { + m_Parent[rootB] = rootA; + } + else + { + m_Parent[rootB] = rootA; + m_Rank[rootA]++; + } + } + + /** + * @brief Add to the size count for a label. + * Sizes are accumulated at each label, not the root. They are + * accumulated to roots during flatten(). + * @param label Label to update + * @param count Number of voxels to add + */ + void addSize(int64 label, uint64 count) + { + ensureCapacity(label); + m_Size[label] += count; + } + + /** + * @brief Get the total size of a label's equivalence class. + * Should only be called after flatten() for accurate totals. + * @param label Label to query + * @return Total number of voxels in the equivalence class + */ + uint64 getSize(int64 label) + { + int64 root = find(label); + return m_Size[root]; + } + + /** + * @brief Flatten the union-find structure with full path compression + * and accumulate all sizes to root labels. + * + * After flatten(): + * - Every label points directly to its root + * - All sizes are accumulated at root labels + * - Subsequent find() calls are O(1) (single lookup) + */ + void flatten() + { + const usize count = m_Parent.size(); + + // Full path compression: point every label directly to its root + for(usize i = 1; i < count; i++) + { + m_Parent[i] = find(static_cast(i)); + } + + // Accumulate sizes to roots + std::vector rootSizes(count, 0); + for(usize i = 1; i < count; i++) + { + rootSizes[m_Parent[i]] += m_Size[i]; + } + m_Size = std::move(rootSizes); + } + +private: + /** + * @brief Ensure the internal vectors can hold index x. + * Grows by doubling to amortize allocation cost. + */ + void ensureCapacity(int64 x) + { + auto idx = static_cast(x); + if(idx < m_Parent.size()) + { + return; + } + + usize newSize = std::max(idx + 1, m_Parent.size() * 2); + usize oldSize = m_Parent.size(); + m_Parent.resize(newSize); + m_Rank.resize(newSize, 0); + m_Size.resize(newSize, 0); + + // Initialize new entries as self-parents + for(usize i = oldSize; i < newSize; i++) + { + m_Parent[i] = static_cast(i); + } + } + + std::vector m_Parent; + std::vector m_Rank; + std::vector m_Size; +}; + +} // namespace nx::core diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp b/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp new file mode 100644 index 0000000000..1649797a30 --- /dev/null +++ b/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp @@ -0,0 +1,632 @@ +#pragma once + +#include "simplnx/DataStructure/AttributeMatrix.hpp" +#include "simplnx/DataStructure/DataArray.hpp" +#include "simplnx/DataStructure/DataStructure.hpp" +#include "simplnx/DataStructure/Geometry/ImageGeom.hpp" +#include "simplnx/Filter/Arguments.hpp" +#include "simplnx/Filter/IFilter.hpp" +#include "simplnx/UnitTest/UnitTestCommon.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" + +#include +#include + +#include +#include +#include +#include +#include + +namespace nx::core::UnitTest +{ + +/** + * @brief Creates an ImageGeom with a CellData AttributeMatrix. + * @param ds DataStructure to create objects in. + * @param dims Voxel dimensions {X, Y, Z}. + * @param geomName Name for the ImageGeom. + * @param cellDataName Name for the CellData AttributeMatrix. + * @return Pointer to the created AttributeMatrix. + */ +inline AttributeMatrix* BuildSegmentFeaturesTestGeometry(DataStructure& ds, const std::array& dims, const std::string& geomName, const std::string& cellDataName) +{ + auto* geom = ImageGeom::Create(ds, geomName); + geom->setDimensions({dims[0], dims[1], dims[2]}); + geom->setSpacing({1.0f, 1.0f, 1.0f}); + geom->setOrigin({0.0f, 0.0f, 0.0f}); + + const ShapeType cellShape = {dims[2], dims[1], dims[0]}; + auto* am = AttributeMatrix::Create(ds, cellDataName, cellShape, geom->getId()); + geom->setCellData(*am); + return am; +} + +/** + * @brief Creates block-patterned int32 scalar data for ScalarSegmentFeatures testing. + * @param ds DataStructure to create the array in. + * @param cellShape Tuple shape {Z, Y, X}. + * @param amId Parent AttributeMatrix ID. + * @param blockSize Voxel count per block edge. + * @param arrayName Name for the scalar array. + */ +inline void BuildScalarTestData(DataStructure& ds, const ShapeType& cellShape, DataObject::IdType amId, usize blockSize, const std::string& arrayName = "ScalarData", bool wrapBoundary = false) +{ + const usize dimZ = cellShape[0]; + const usize dimY = cellShape[1]; + const usize dimX = cellShape[2]; + + auto scalarDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* scalarArray = DataArray::Create(ds, arrayName, scalarDataStore, amId); + auto& store = scalarArray->getDataStoreRef(); + + const usize blocksPerX = (dimX + blockSize - 1) / blockSize; + const usize blocksPerY = (dimY + blockSize - 1) / blockSize; + const usize blocksPerZ = (dimZ + blockSize - 1) / blockSize; + + const usize sliceSize = dimY * dimX; + std::vector sliceBuffer(sliceSize); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize bx = x / blockSize; + const usize by = y / blockSize; + const usize bz = z / blockSize; + + if(wrapBoundary) + { + // Last block in each axis maps to the same value as the first block, + // so periodic wrapping merges them into one feature. + const usize wbx = (bx == blocksPerX - 1) ? 0 : bx; + const usize wby = (by == blocksPerY - 1) ? 0 : by; + const usize wbz = (bz == blocksPerZ - 1) ? 0 : bz; + const usize wbpx = blocksPerX - 1; + const usize wbpy = blocksPerY - 1; + sliceBuffer[y * dimX + x] = static_cast(wbz * wbpy * wbpx + wby * wbpx + wbx); + } + else + { + sliceBuffer[y * dimX + x] = static_cast(bz * blocksPerY * blocksPerX + by * blocksPerX + bx); + } + } + } + store.copyFromBuffer(z * sliceSize, nonstd::span(sliceBuffer.data(), sliceSize)); + } +} + +/** + * @brief Creates quaternion, phase, and crystal structure arrays for EBSD/CAxis testing. + * + * Quaternions are block-patterned with distinct orientations per block. + * All voxels are assigned phase 1. CrystalStructures has phase 0 = 999 (Unknown) + * and phase 1 = the provided crystal structure value. + * + * @param ds DataStructure to create arrays in. + * @param cellShape Tuple shape {Z, Y, X}. + * @param geomId Parent geometry ID (for ensemble AM). + * @param amId Parent CellData AttributeMatrix ID. + * @param crystalStructure Crystal structure for phase 1 (1 = Cubic_High, 0 = Hexagonal_High). + * @param blockSize Voxel count per block edge. + */ +inline void BuildOrientationTestData(DataStructure& ds, const ShapeType& cellShape, DataObject::IdType geomId, DataObject::IdType amId, uint32 crystalStructure, usize blockSize, + bool wrapBoundary = false) +{ + const usize dimZ = cellShape[0]; + const usize dimY = cellShape[1]; + const usize dimX = cellShape[2]; + + auto quatsDataStore = DataStoreUtilities::CreateDataStore(cellShape, {4}, IDataAction::Mode::Execute); + auto* quatsArray = DataArray::Create(ds, "Quats", quatsDataStore, amId); + auto& quatsStore = quatsArray->getDataStoreRef(); + + auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* phasesArray = DataArray::Create(ds, "Phases", phasesDataStore, amId); + auto& phasesStore = phasesArray->getDataStoreRef(); + + constexpr float32 k_DegToRad = 3.14159265358979323846f / 180.0f; + + const usize blocksPerX = (dimX + blockSize - 1) / blockSize; + const usize blocksPerY = (dimY + blockSize - 1) / blockSize; + const usize blocksPerZ = (dimZ + blockSize - 1) / blockSize; + const usize numBlocks = blocksPerX * blocksPerY * blocksPerZ; + + // Quaternion Hamilton product: result = a * b, where q = (w, x, y, z) + auto quatMul = [](const std::array& a, const std::array& b) -> std::array { + return {a[0] * b[0] - a[1] * b[1] - a[2] * b[2] - a[3] * b[3], a[0] * b[1] + a[1] * b[0] + a[2] * b[3] - a[3] * b[2], a[0] * b[2] - a[1] * b[3] + a[2] * b[0] + a[3] * b[1], + a[0] * b[3] + a[1] * b[2] - a[2] * b[1] + a[3] * b[0]}; + }; + + std::vector> blockQuats(numBlocks); + + // Z-layer orientation scheme (shared by EBSD and CAxis): + // All blocks in the same Z-layer share a single X-axis rotation angle. + // This produces 3 horizontal layers of identical orientations: + // z=0: 0° rotation → q = [1, 0, 0, 0] c-axis = [0, 0, 1] + // z=1: 30° rotation → q = [0.966, 0.259, 0, 0] c-axis = [0, 0.5, 0.866] + // z=2: 60° rotation → q = [0.866, 0.5, 0, 0] c-axis = [0, 0.866, 0.5] + // + // Adjacent layers differ by 30°, well above the 5° tolerance → no merge. + // Within each layer, all blocks share the same angle → they merge. + // + // Merge pair override (non-periodic only): + // Block (1,1,1) at center of z=1 is set to 0° instead of 30°. + // It merges with its z=0 neighbor (1,1,0) while staying separate + // from the other z=1 blocks (30° difference → no merge). + // + // Expected features (3x3x3 grid): + // Base (3 blocks/axis): 3 features (z=0 + center pillar, z=1 minus pillar, z=2) + // Base (8 blocks/axis): 3 features (repeating 0°/30°/60° stripes) + // Periodic: layers sharing the same angle merge across the boundary + constexpr float32 k_LayerAngles[] = {0.0f, 30.0f, 60.0f}; + + for(usize bz = 0; bz < blocksPerZ; bz++) + { + const usize layerIdx = bz % 3; + const float32 halfAngle = k_LayerAngles[layerIdx] * k_DegToRad * 0.5f; + // EBSDlib quaternion layout: (x, y, z, w) — Vector-Scalar order + const std::array layerQuat = {std::sin(halfAngle), 0.0f, 0.0f, std::cos(halfAngle)}; + + for(usize by = 0; by < blocksPerY; by++) + { + for(usize bx = 0; bx < blocksPerX; bx++) + { + const usize blockIdx = bz * blocksPerY * blocksPerX + by * blocksPerX + bx; + blockQuats[blockIdx] = layerQuat; + } + } + } + + // Merge pair: block (1,1,1) gets z=0 angle (0°) instead of z=1 angle (30°). + // It merges downward into the z=0 layer through face neighbor (1,1,0). + if(!wrapBoundary && blocksPerX >= 3 && blocksPerY >= 3 && blocksPerZ >= 3) + { + const usize idx_111 = 1 * blocksPerY * blocksPerX + 1 * blocksPerX + 1; + blockQuats[idx_111] = blockQuats[0]; // Set to 0° (z=0 layer angle) + } + + const usize sliceSize = dimY * dimX; + std::vector quatsSliceBuffer(sliceSize * 4); + std::vector phasesSliceBuffer(sliceSize, 1); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + usize bx = x / blockSize; + usize by = y / blockSize; + usize bz = z / blockSize; + if(wrapBoundary) + { + bx = (bx == blocksPerX - 1) ? 0 : bx; + by = (by == blocksPerY - 1) ? 0 : by; + bz = (bz == blocksPerZ - 1) ? 0 : bz; + } + const usize blockIdx = bz * blocksPerY * blocksPerX + by * blocksPerX + bx; + const auto& q = blockQuats[blockIdx]; + const usize bufIdx = (y * dimX + x) * 4; + quatsSliceBuffer[bufIdx + 0] = q[0]; + quatsSliceBuffer[bufIdx + 1] = q[1]; + quatsSliceBuffer[bufIdx + 2] = q[2]; + quatsSliceBuffer[bufIdx + 3] = q[3]; + } + } + quatsStore.copyFromBuffer(z * sliceSize * 4, nonstd::span(quatsSliceBuffer.data(), sliceSize * 4)); + phasesStore.copyFromBuffer(z * sliceSize, nonstd::span(phasesSliceBuffer.data(), sliceSize)); + } + + // Create CellEnsembleData with CrystalStructures + const ShapeType ensembleTupleShape = {2}; + auto* ensembleAM = AttributeMatrix::Create(ds, "CellEnsembleData", ensembleTupleShape, geomId); + auto crystalDataStore = DataStoreUtilities::CreateDataStore(ensembleTupleShape, {1}, IDataAction::Mode::Execute); + auto* crystalStructsArray = DataArray::Create(ds, "CrystalStructures", crystalDataStore, ensembleAM->getId()); + auto& crystalStructsStore = crystalStructsArray->getDataStoreRef(); + crystalStructsStore[0] = 999; // Phase 0: Unknown + crystalStructsStore[1] = crystalStructure; +} + +/** + * @brief Creates a spherical mask array where voxels inside the sphere are 1 (good) + * and voxels outside are 0 (masked out). + * + * The sphere is centered in the volume with radius = 80% of half the smallest dimension. + * For a 200x200x200 volume, that gives a radius of 80 voxels. + * + * @param ds DataStructure to create the array in. + * @param cellShape Tuple shape {Z, Y, X}. + * @param amId Parent AttributeMatrix ID. + * @param maskName Name for the mask array. + */ +inline void BuildSphericalMask(DataStructure& ds, const ShapeType& cellShape, DataObject::IdType amId, const std::string& maskName = "Mask") +{ + const usize dimZ = cellShape[0]; + const usize dimY = cellShape[1]; + const usize dimX = cellShape[2]; + + auto maskDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto* maskArray = DataArray::Create(ds, maskName, maskDataStore, amId); + auto& maskStore = maskArray->getDataStoreRef(); + + const float32 cx = static_cast(dimX) / 2.0f; + const float32 cy = static_cast(dimY) / 2.0f; + const float32 cz = static_cast(dimZ) / 2.0f; + const float32 radius = std::min({cx, cy, cz}) * 0.8f; + + const usize sliceSize = dimY * dimX; + std::vector sliceBuffer(sliceSize); + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const float32 dx = static_cast(x) - cx; + const float32 dy = static_cast(y) - cy; + const float32 dz = static_cast(z) - cz; + sliceBuffer[y * dimX + x] = (dx * dx + dy * dy + dz * dz < radius * radius) ? 1 : 0; + } + } + maskStore.copyFromBuffer(z * sliceSize, nonstd::span(sliceBuffer.data(), sliceSize)); + } +} + +/** + * @brief Verifies segmentation results when a mask is applied. + * + * Checks that: + * 1. Masked voxels (mask=0) have FeatureId=0 + * 2. Unmasked voxels (mask=1) have FeatureId > 0 + * 3. At least one feature was created + * 4. Both masked and unmasked regions exist + * + * @param ds DataStructure containing the results. + * @param dims Voxel dimensions {X, Y, Z}. + * @param featureIdsPath Path to the generated FeatureIds array. + * @param activePath Path to the generated Active array. + * @param maskPath Path to the mask array. + */ +inline void VerifyMaskedSegmentation(const DataStructure& ds, const std::array& dims, const DataPath& featureIdsPath, const DataPath& activePath, const DataPath& maskPath) +{ + REQUIRE_NOTHROW(ds.getDataRefAs(featureIdsPath)); + const auto& featureIds = ds.getDataRefAs(featureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + + REQUIRE_NOTHROW(ds.getDataRefAs(maskPath)); + const auto& mask = ds.getDataRefAs(maskPath); + const auto& maskStore = mask.getDataStoreRef(); + + REQUIRE_NOTHROW(ds.getDataRefAs(activePath)); + const auto& actives = ds.getDataRefAs(activePath); + REQUIRE(actives.getNumberOfTuples() > 1); // At least one feature (index 0 + features) + + const usize totalVoxels = dims[0] * dims[1] * dims[2]; + usize maskedCount = 0; + usize unmaskedCount = 0; + + for(usize i = 0; i < totalVoxels; i++) + { + if(maskStore.getValue(i) == 0) + { + REQUIRE(featureStore.getValue(i) == 0); + maskedCount++; + } + else + { + REQUIRE(featureStore.getValue(i) > 0); + unmaskedCount++; + } + } + + REQUIRE(maskedCount > 0); + REQUIRE(unmaskedCount > 0); +} + +/** + * @brief Verifies that block-patterned segmentation produced the expected results. + * + * Checks that: + * 1. The feature count matches the expected number of blocks + * 2. All voxels within a block share the same FeatureId + * 3. Different blocks have different FeatureIds + * + * @param ds DataStructure containing the results. + * @param dims Voxel dimensions {X, Y, Z}. + * @param blockSize Voxel count per block edge. + * @param featureIdsPath Path to the generated FeatureIds array. + * @param activePath Path to the generated Active array. + */ +inline void VerifyBlockSegmentation(const DataStructure& ds, const std::array& dims, usize blockSize, const DataPath& featureIdsPath, const DataPath& activePath) +{ + const usize dimX = dims[0]; + const usize dimY = dims[1]; + const usize dimZ = dims[2]; + const usize blocksPerX = (dimX + blockSize - 1) / blockSize; + const usize blocksPerY = (dimY + blockSize - 1) / blockSize; + const usize blocksPerZ = (dimZ + blockSize - 1) / blockSize; + const usize expectedFeatures = blocksPerX * blocksPerY * blocksPerZ; + + // Check feature count (Active array includes Feature 0) + REQUIRE_NOTHROW(ds.getDataRefAs(activePath)); + const auto& actives = ds.getDataRefAs(activePath); + REQUIRE(actives.getNumberOfTuples() == expectedFeatures + 1); + + // Check FeatureIds consistency + REQUIRE_NOTHROW(ds.getDataRefAs(featureIdsPath)); + const auto& featureIds = ds.getDataRefAs(featureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + + // Map from block index to the FeatureId assigned to that block + std::unordered_map blockToFeature; + // Track all assigned FeatureIds to verify uniqueness + std::set usedFeatureIds; + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize voxelIdx = z * dimX * dimY + y * dimX + x; + const usize blockIdx = (z / blockSize) * blocksPerY * blocksPerX + (y / blockSize) * blocksPerX + (x / blockSize); + const int32 featureId = featureStore.getValue(voxelIdx); + + REQUIRE(featureId > 0); // No voxel should be unassigned + + auto it = blockToFeature.find(blockIdx); + if(it == blockToFeature.end()) + { + blockToFeature[blockIdx] = featureId; + usedFeatureIds.insert(featureId); + } + else + { + REQUIRE(it->second == featureId); // All voxels in a block share the same FeatureId + } + } + } + } + + // Each block should have a unique FeatureId + REQUIRE(usedFeatureIds.size() == expectedFeatures); +} + +/** + * @brief Verifies segmentation results when periodic BCs are enabled and boundary + * blocks have matching data (wrapBoundary=true). + * + * With periodic wrapping, the last block in each axis merges with the first block. + * Expected feature count: (blocksPerX-1) * (blocksPerY-1) * (blocksPerZ-1). + * + * @param ds DataStructure containing the results. + * @param dims Voxel dimensions {X, Y, Z}. + * @param blockSize Voxel count per block edge. + * @param featureIdsPath Path to the generated FeatureIds array. + * @param activePath Path to the generated Active array. + */ +inline void VerifyPeriodicBlockSegmentation(const DataStructure& ds, const std::array& dims, usize blockSize, const DataPath& featureIdsPath, const DataPath& activePath) +{ + const usize dimX = dims[0]; + const usize dimY = dims[1]; + const usize dimZ = dims[2]; + const usize blocksPerX = (dimX + blockSize - 1) / blockSize; + const usize blocksPerY = (dimY + blockSize - 1) / blockSize; + const usize blocksPerZ = (dimZ + blockSize - 1) / blockSize; + const usize periodicBlocksX = blocksPerX - 1; + const usize periodicBlocksY = blocksPerY - 1; + const usize periodicBlocksZ = blocksPerZ - 1; + const usize expectedFeatures = periodicBlocksX * periodicBlocksY * periodicBlocksZ; + + // Check feature count (Active array includes Feature 0) + REQUIRE_NOTHROW(ds.getDataRefAs(activePath)); + const auto& actives = ds.getDataRefAs(activePath); + REQUIRE(actives.getNumberOfTuples() == expectedFeatures + 1); + + // Check FeatureIds consistency + REQUIRE_NOTHROW(ds.getDataRefAs(featureIdsPath)); + const auto& featureIds = ds.getDataRefAs(featureIdsPath); + const auto& featureStore = featureIds.getDataStoreRef(); + + // Map from periodic block index to the FeatureId assigned to that block + std::unordered_map blockToFeature; + std::set usedFeatureIds; + + for(usize z = 0; z < dimZ; z++) + { + for(usize y = 0; y < dimY; y++) + { + for(usize x = 0; x < dimX; x++) + { + const usize voxelIdx = z * dimX * dimY + y * dimX + x; + const usize bx = x / blockSize; + const usize by = y / blockSize; + const usize bz = z / blockSize; + + // Effective periodic block index: last block wraps to first + const usize pbx = bx % periodicBlocksX; + const usize pby = by % periodicBlocksY; + const usize pbz = bz % periodicBlocksZ; + const usize periodicBlockIdx = pbz * periodicBlocksY * periodicBlocksX + pby * periodicBlocksX + pbx; + + const int32 featureId = featureStore.getValue(voxelIdx); + REQUIRE(featureId > 0); // No voxel should be unassigned + + auto it = blockToFeature.find(periodicBlockIdx); + if(it == blockToFeature.end()) + { + blockToFeature[periodicBlockIdx] = featureId; + usedFeatureIds.insert(featureId); + } + else + { + REQUIRE(it->second == featureId); // All voxels in matching periodic blocks share the same FeatureId + } + } + } + } + + // Each periodic block group should have a unique FeatureId + REQUIRE(usedFeatureIds.size() == expectedFeatures); +} + +/** + * @brief Runs the "no valid voxels returns error -87000" test for any SegmentFeatures filter. + * + * Creates a 3x3x3 grid with all voxels masked out, runs the filter, and asserts + * that execution returns error -87000. + * + * @tparam FilterT The filter class (e.g., ScalarSegmentFeaturesFilter). + * @param setupArgs Lambda that receives (Arguments&, DataPath geomPath, DataPath cellDataPath, DataPath maskPath) + * and inserts filter-specific arguments. + */ +template +void RunNoValidVoxelsErrorTest(SetupArgsFn setupArgs) +{ + constexpr usize kDim = 3; + const std::array dims = {kDim, kDim, kDim}; + const ShapeType cellShape = {kDim, kDim, kDim}; + + DataStructure ds; + auto* am = BuildSegmentFeaturesTestGeometry(ds, dims, "Geom", "CellData"); + + auto* mask = CreateTestDataArray(ds, "Mask", cellShape, {1}, am->getId()); + mask->fill(0); + + const DataPath geomPath({"Geom"}); + const DataPath cellDataPath({"Geom", "CellData"}); + const DataPath maskPath({"Geom", "CellData", "Mask"}); + + FilterT filter; + Arguments args; + setupArgs(args, ds, geomPath, cellDataPath, maskPath); + + auto preflightResult = filter.preflight(ds, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(ds, args); + SIMPLNX_RESULT_REQUIRE_INVALID(executeResult.result); + + const auto& errors = executeResult.result.errors(); + REQUIRE(errors.size() == 1); + REQUIRE(errors[0].code == -87000); +} + +/** + * @brief Tests that FaceEdgeVertex (26-neighbor) connectivity correctly merges + * regions connected through shared vertices and edges, not just faces. + * + * Creates a 3x3x3 geometry with 4 isolated single-voxel regions: + * - Regions A,B (same data): voxels (0,0,0) and (1,1,1) — vertex-connected only + * - Regions C,D (same data): voxels (2,0,0) and (2,1,1) — edge-connected only + * + * With Face (6-neighbor): 5 features (1 background + 4 isolated regions) + * With FaceEdgeVertex (26-neighbor): 3 features (1 background + A&B merged + C&D merged) + * + * @tparam FilterT The filter class (e.g., ScalarSegmentFeaturesFilter). + * @param setupFaceArgs Lambda (Arguments&, DataStructure&, DataPath geomPath, DataPath cellDataPath) + * that inserts filter-specific arguments with neighborScheme=0 (Face). + * @param setupFevArgs Lambda with same signature but neighborScheme=1 (FaceEdgeVertex). + */ +template +void RunFaceEdgeVertexConnectivityTest(SetupFaceFn setupFaceArgs, SetupFevFn setupFevArgs) +{ + constexpr usize kDim = 3; + const std::array dims = {kDim, kDim, kDim}; + const ShapeType cellShape = {kDim, kDim, kDim}; + + const DataPath geomPath({"Geom"}); + const DataPath cellDataPath({"Geom", "CellData"}); + const DataPath featureIdsPath({"Geom", "CellData", "FeatureIds"}); + const DataPath activePath({"Geom", "CellFeatureData", "Active"}); + + // Face scheme: A, B, C, D are all isolated → 5 features + index 0 + { + DataStructure ds; + BuildSegmentFeaturesTestGeometry(ds, dims, "Geom", "CellData"); + FilterT filter; + Arguments args; + setupFaceArgs(args, ds, geomPath, cellDataPath); + auto preflightResult = filter.preflight(ds, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(ds, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + const auto& actives = ds.getDataRefAs(activePath); + REQUIRE(actives.getNumberOfTuples() == 6); + } + + // FaceEdgeVertex scheme: A+B merge (vertex), C+D merge (edge) → 3 features + index 0 + DataStructure ds; + BuildSegmentFeaturesTestGeometry(ds, dims, "Geom", "CellData"); + { + FilterT filter; + Arguments args; + setupFevArgs(args, ds, geomPath, cellDataPath); + auto preflightResult = filter.preflight(ds, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + auto executeResult = filter.execute(ds, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + const auto& actives = ds.getDataRefAs(activePath); + REQUIRE(actives.getNumberOfTuples() == 4); + } + + // Verify the vertex-connected pair shares a FeatureId + const auto& fids = ds.getDataRefAs(featureIdsPath); + const auto& fidsStore = fids.getDataStoreRef(); + REQUIRE(fidsStore.getValue(0 * 9 + 0 * 3 + 0) == fidsStore.getValue(1 * 9 + 1 * 3 + 1)); // A == B (vertex merge) + REQUIRE(fidsStore.getValue(0 * 9 + 0 * 3 + 2) == fidsStore.getValue(1 * 9 + 1 * 3 + 2)); // C == D (edge merge) + REQUIRE(fidsStore.getValue(0 * 9 + 0 * 3 + 0) != fidsStore.getValue(0 * 9 + 0 * 3 + 2)); // A != C (different values) +} + +/** + * @brief Runs a SegmentFeatures filter against exemplar data and verifies results. + * + * Executes the filter, optionally checks the feature count, compares computed + * FeatureIds against embedded exemplar arrays, and validates tuple dimension + * inheritance. Used by Scalar, EBSD, and CAxis neighbor scheme tests. + * + * @tparam FilterT The filter class (e.g., ScalarSegmentFeaturesFilter). + * @tparam SetupArgsFn Lambda (Arguments&) that inserts all filter-specific arguments. + * @param dataStructure DataStructure loaded from an exemplar .dream3d file. + * @param computedFeatureIdsPath Path where the filter writes its FeatureIds array. + * @param activesPath Path where the filter writes its Active array. + * @param exemplarFeatureIdsPath Path to the pre-computed exemplar FeatureIds. + * @param expectedFeatureCount Expected Active tuple count (0 to skip this check). + * @param setupArgs Lambda to populate filter Arguments. + * @param tupleCheckIgnoredPaths Paths to exclude from CheckArraysInheritTupleDims. + */ +template +void RunNeighborSchemeExemplarTest(DataStructure& dataStructure, const DataPath& computedFeatureIdsPath, const DataPath& activesPath, const DataPath& exemplarFeatureIdsPath, + usize expectedFeatureCount, SetupArgsFn setupArgs, const std::vector& tupleCheckIgnoredPaths = {}) +{ + FilterT filter; + Arguments args; + setupArgs(args); + + auto preflightResult = filter.preflight(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(preflightResult.outputActions); + + auto executeResult = filter.execute(dataStructure, args); + SIMPLNX_RESULT_REQUIRE_VALID(executeResult.result); + + if(expectedFeatureCount > 0) + { + REQUIRE_NOTHROW(dataStructure.getDataRefAs(activesPath)); + const auto& actives = dataStructure.getDataRefAs(activesPath); + REQUIRE(actives.getNumberOfTuples() == expectedFeatureCount); + } + + REQUIRE_NOTHROW(dataStructure.getDataRefAs(computedFeatureIdsPath)); + REQUIRE_NOTHROW(dataStructure.getDataRefAs(exemplarFeatureIdsPath)); + const auto& generatedArray = dataStructure.getDataRefAs(computedFeatureIdsPath); + const auto& exemplarArray = dataStructure.getDataRefAs(exemplarFeatureIdsPath); + CompareDataArrays(generatedArray, exemplarArray); + + CheckArraysInheritTupleDims(dataStructure, tupleCheckIgnoredPaths); +} + +} // namespace nx::core::UnitTest diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp index f609439399..c1ee31e7a3 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp @@ -34,6 +34,7 @@ #include #include +#include #include #include @@ -220,7 +221,7 @@ const DataPath k_ExemplarDataContainerPath({k_ExemplarDataContainer}); namespace UnitTest { -inline constexpr float EPSILON = 0.0001; +inline constexpr float32 EPSILON = 0.0001; template std::string ComputeMD5Hash(const std::vector& outputDataArray) @@ -241,16 +242,18 @@ class TestFileSentinel { public: /** - * @brief Construct a File Sentinel object that will decompress on construction and remove the - * contents on destruction. + * @brief Construct a File Sentinel object that will decompress on construction if the + * expected top-level output does not already exist, and remove the decompressed + * contents on destruction only when no other sentinel still references the same archive. * * @param testFilesDir The directory where the archive is located * @param inputArchiveName The full name of the archive. The location is assumed to be in the TestFiles directory * @param expectedTopLevelOutput The name of the decompressed folder or file. WARNING: This assumes * that only a single file or single directory are part of the archive. In the case of a directory, the * directory itself can have as many subdirectories as needed. - * @param decompressFiles Decompress the archive - * @param removeTemp delete files that were decompressed + * @param decompressFiles Decompress the archive only if the expected top-level output does not already exist + * @param removeTemp delete the decompressed contents on destruction (only when this is the last + * sentinel referencing the archive, to avoid deleting data while parallel tests still read from it) */ TestFileSentinel(std::string testFilesDir, std::string inputArchiveName, std::string expectedTopLevelOutput, bool decompressFiles = true, bool removeTemp = true); @@ -273,6 +276,7 @@ class TestFileSentinel std::string m_ExpectedTopLevelOutput; bool m_Decompress; bool m_RemoveTemp; + std::filesystem::path m_HolderFile; // Per-process holder file inside {expected}.holders/ }; /** @@ -306,6 +310,38 @@ class PreferencesSentinel bool m_OriginalForceOoc; }; +/** + * @brief Returns the expected IDataStore::StoreType based on current preferences + * and whether the OOC plugin is loaded. When forceOocData is true AND the + * large data format is registered, expects OutOfCore. Otherwise expects InMemory. + */ +inline IDataStore::StoreType ExpectedStoreType() +{ + auto* prefs = Application::GetOrCreateInstance()->getPreferences(); + if(prefs->forceOocData()) + { + auto ioCollection = DataStoreUtilities::GetIOCollection(); + auto manager = ioCollection->getManager(prefs->largeDataFormat()); + if(manager != nullptr) + { + return IDataStore::StoreType::OutOfCore; + } + } + return IDataStore::StoreType::InMemory; +} + +/** + * @brief REQUIREs that the given data array's store type matches what is expected + * based on the current preferences and plugin state. + */ +inline void RequireExpectedStoreType(const IDataArray& array) +{ + auto expected = ExpectedStoreType(); + auto actual = array.getIDataStoreRef().getStoreType(); + INFO("Array '" << array.getName() << "' StoreType: expected=" << static_cast(expected) << " actual=" << static_cast(actual)); + REQUIRE(actual == expected); +} + /** * @brief closeEnough * @param a @@ -457,34 +493,16 @@ inline void CompareMontage(const AbstractMontage& exemplar, const AbstractMontag } /** - * @brief Compares two IDataArrays element-by-element using bulk copyIntoBuffer - * for OOC-safe, high-performance comparison. - * - * Why copyIntoBuffer instead of operator[]: - * When arrays are backed by an out-of-core (chunked) DataStore, each call - * to operator[] may trigger a chunk load from disk. Comparing millions of - * elements one at a time would cause catastrophic chunk thrashing. Instead, - * this function reads both arrays in 40,000-element chunks via copyIntoBuffer, - * which batches HDF5 I/O and keeps access sequential. This is also safe for - * in-memory stores, where copyIntoBuffer is a simple memcpy. + * @brief Compares two IDataArrays using bulk copyIntoBuffer for OOC efficiency. * - * Floating-point comparison semantics: - * - NaN == NaN is treated as equal. Many filter outputs legitimately produce - * NaN values (e.g., division by zero in optional statistics), and both the - * exemplar and generated arrays should agree on which elements are NaN. - * - Values within UnitTest::EPSILON of each other are treated as equal, - * accommodating floating-point rounding differences across platforms. + * Reads both arrays in 40K-element chunks via copyIntoBuffer instead of + * per-element operator[], avoiding per-voxel OOC store overhead. + * NaN == NaN is treated as equal for floating-point types. * - * Error reporting: - * On the first mismatched element, the function records the index and both - * values, then breaks out of the comparison loop. The mismatch details are - * reported via Catch2's UNSCOPED_INFO before the final REQUIRE(!failed). - * - * @tparam T The element type (must match the actual DataStore element type) - * @param left First array (typically the exemplar / golden reference) - * @param right Second array (typically the generated / computed result) - * @param start Element index to start comparison from (default 0). Useful when - * the first N elements are known to differ (e.g., header/padding). + * @tparam T The element type + * @param left First array (typically exemplar) + * @param right Second array (typically generated) + * @param start Element index to start comparison from (default 0) */ template void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize start = 0) @@ -495,9 +513,6 @@ void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize st INFO(fmt::format("Input Data Array:'{}' Output DataArray: '{}' bad comparison", left.getName(), right.getName())); REQUIRE(totalSize == newDataStore.getSize()); - // Use 40K-element chunks to balance memory usage against I/O efficiency. - // Each chunk is ~160 KB for float32 or ~320 KB for float64, which fits - // comfortably in L2 cache and aligns well with typical HDF5 chunk sizes. constexpr usize k_ChunkSize = 40000; auto oldBuf = std::make_unique(k_ChunkSize); auto newBuf = std::make_unique(k_ChunkSize); @@ -507,16 +522,12 @@ void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize st T failOld = {}; T failNew = {}; - // Iterate through the arrays in fixed-size chunks, reading both arrays - // into local buffers for fast element-wise comparison for(usize offset = start; offset < totalSize && !failed; offset += k_ChunkSize) { - // Handle the last chunk which may be smaller than k_ChunkSize const usize count = std::min(k_ChunkSize, totalSize - offset); oldDataStore.copyIntoBuffer(offset, nonstd::span(oldBuf.get(), count)); newDataStore.copyIntoBuffer(offset, nonstd::span(newBuf.get(), count)); - // Compare each element in the current chunk for(usize i = 0; i < count; i++) { const T oldVal = oldBuf[i]; @@ -525,20 +536,17 @@ void CompareDataArrays(const IDataArray& left, const IDataArray& right, usize st { if constexpr(std::is_floating_point_v) { - // Special case: NaN == NaN is treated as equal because many filters - // produce NaN for undefined results, and both arrays should agree + // NaN == NaN is treated as equal if(std::isnan(oldVal) && std::isnan(newVal)) { continue; } - // Allow small floating-point differences within EPSILON tolerance float32 diff = std::fabs(static_cast(oldVal - newVal)); if(diff <= EPSILON) { continue; } } - // Record the first failure for diagnostic output, then stop failed = true; failIndex = offset + i; failOld = oldVal; @@ -585,33 +593,51 @@ void CompareDataArraysByComponent(const IDataArray& left, const IDataArray& righ INFO(fmt::format("Input Data Array:'{}' Output DataArray: '{}' bad comparison", left.getName(), right.getName())); REQUIRE(startTuple < tupleCount); REQUIRE(component < componentCount); - T oldVal; - T newVal; + + constexpr usize k_ChunkTuples = 40000; + const usize chunkElements = k_ChunkTuples * componentCount; + auto oldBuf = std::make_unique(chunkElements); + auto newBuf = std::make_unique(chunkElements); + bool failed = false; - for(usize t = startTuple; t < tupleCount; t++) + usize failTuple = 0; + T failOld = {}; + T failNew = {}; + + for(usize tStart = startTuple; tStart < tupleCount && !failed; tStart += k_ChunkTuples) { - oldVal = oldDataStore[t * componentCount + component]; - newVal = newDataStore[t * componentCount + component]; - if(oldVal != newVal) - { - UNSCOPED_INFO(fmt::format("tuple=: {} component=: {} oldValue != newValue. {} != {}", t, component, oldVal, newVal)); + const usize tCount = std::min(k_ChunkTuples, tupleCount - tStart); + const usize elemCount = tCount * componentCount; + oldDataStore.copyIntoBuffer(tStart * componentCount, nonstd::span(oldBuf.get(), elemCount)); + newDataStore.copyIntoBuffer(tStart * componentCount, nonstd::span(newBuf.get(), elemCount)); - if constexpr(std::is_floating_point_v) + for(usize t = 0; t < tCount; t++) + { + const T oldVal = oldBuf[t * componentCount + component]; + const T newVal = newBuf[t * componentCount + component]; + if(oldVal != newVal) { - float diff = std::fabs(static_cast(oldVal - newVal)); - if(diff > EPSILON) + if constexpr(std::is_floating_point_v) { - failed = true; - break; + float32 diff = std::fabs(static_cast(oldVal - newVal)); + if(diff <= EPSILON) + { + continue; + } } - } - else - { failed = true; + failTuple = tStart + t; + failOld = oldVal; + failNew = newVal; + break; } - break; } } + + if(failed) + { + UNSCOPED_INFO(fmt::format("tuple=: {} component=: {} oldValue != newValue. {} != {}", failTuple, component, failOld, failNew)); + } REQUIRE(!failed); } @@ -625,7 +651,6 @@ void CompareDataArraysByComponent(const IDataArray& left, const IDataArray& righ template void CompareArrays(const DataStructure& dataStructure, const DataPath& exemplaryDataPath, const DataPath& computedPath) { - // DataPath exemplaryDataPath = featureGroup.createChildPath("SurfaceFeatures"); REQUIRE_NOTHROW(dataStructure.getDataRefAs>(exemplaryDataPath)); REQUIRE_NOTHROW(dataStructure.getDataRefAs>(computedPath)); INFO(fmt::format("Exemplary Data Array:'{}'\n Computed DataArray: '{}'\n bad comparison", exemplaryDataPath.toString(), computedPath.toString())); @@ -634,19 +659,47 @@ void CompareArrays(const DataStructure& dataStructure, const DataPath& exemplary const auto& computedDataArray = dataStructure.getDataRefAs>(computedPath); REQUIRE(exemplaryDataArray.getNumberOfTuples() == computedDataArray.getNumberOfTuples()); - usize start = 0; - usize end = exemplaryDataArray.getSize(); - for(usize i = start; i < end; i++) + const auto& oldStore = exemplaryDataArray.getDataStoreRef(); + const auto& newStore = computedDataArray.getDataStoreRef(); + const usize totalSize = oldStore.getSize(); + + constexpr usize k_ChunkSize = 40000; + auto oldBuf = std::make_unique(k_ChunkSize); + auto newBuf = std::make_unique(k_ChunkSize); + + bool failed = false; + usize failIndex = 0; + T failOld = {}; + T failNew = {}; + + for(usize offset = 0; offset < totalSize && !failed; offset += k_ChunkSize) { - auto oldVal = exemplaryDataArray[i]; - auto newVal = computedDataArray[i]; - if(oldVal != newVal) + const usize count = std::min(k_ChunkSize, totalSize - offset); + oldStore.copyIntoBuffer(offset, nonstd::span(oldBuf.get(), count)); + newStore.copyIntoBuffer(offset, nonstd::span(newBuf.get(), count)); + + for(usize i = 0; i < count; i++) { - float diff = std::fabs(static_cast(oldVal - newVal)); - REQUIRE(diff < EPSILON); - break; + if(oldBuf[i] != newBuf[i]) + { + float32 diff = std::fabs(static_cast(oldBuf[i] - newBuf[i])); + if(diff >= EPSILON) + { + failed = true; + failIndex = offset + i; + failOld = oldBuf[i]; + failNew = newBuf[i]; + break; + } + } } } + + if(failed) + { + UNSCOPED_INFO(fmt::format("index=: {} oldValue != newValue. {} != {}", failIndex, failOld, failNew)); + } + REQUIRE(!failed); } /** @@ -670,28 +723,57 @@ void CompareFloatArraysWithNans(const DataStructure& dataStructure, const DataPa INFO(fmt::format("Input Data Array:'{}' Output DataArray: '{}' bad comparison", exemplaryDataPath.toString(), computedPath.toString())); - usize start = 0; - usize end = exemplaryDataArray.getSize(); - for(usize i = start; i < end; i++) + const auto& oldStore = exemplaryDataArray.getDataStoreRef(); + const auto& newStore = generatedDataArray.getDataStoreRef(); + const usize totalSize = oldStore.getSize(); + + constexpr usize k_ChunkSize = 40000; + auto oldBuf = std::make_unique(k_ChunkSize); + auto newBuf = std::make_unique(k_ChunkSize); + + bool failed = false; + usize failIndex = 0; + T failOld = {}; + T failNew = {}; + + for(usize offset = 0; offset < totalSize && !failed; offset += k_ChunkSize) { - auto oldVal = exemplaryDataArray[i]; - auto newVal = generatedDataArray[i]; - if(!checkNans && (std::isnan(newVal) || std::isnan(oldVal))) - { - continue; - } - if(std::isnan(oldVal) && std::isnan(newVal)) - { - // https://stackoverflow.com/questions/38798791/nan-comparison-rule-in-c-c - continue; - } - if(oldVal != newVal) + const usize count = std::min(k_ChunkSize, totalSize - offset); + oldStore.copyIntoBuffer(offset, nonstd::span(oldBuf.get(), count)); + newStore.copyIntoBuffer(offset, nonstd::span(newBuf.get(), count)); + + for(usize i = 0; i < count; i++) { - float diff = std::fabs(static_cast(oldVal - newVal)); - REQUIRE(diff < epsilon); - break; + const T oldVal = oldBuf[i]; + const T newVal = newBuf[i]; + if(!checkNans && (std::isnan(newVal) || std::isnan(oldVal))) + { + continue; + } + if(std::isnan(oldVal) && std::isnan(newVal)) + { + continue; + } + if(oldVal != newVal) + { + float32 diff = std::fabs(static_cast(oldVal - newVal)); + if(diff >= epsilon) + { + failed = true; + failIndex = offset + i; + failOld = oldVal; + failNew = newVal; + break; + } + } } } + + if(failed) + { + UNSCOPED_INFO(fmt::format("index=: {} oldValue != newValue. {} != {}", failIndex, failOld, failNew)); + } + REQUIRE(!failed); } /** @@ -735,7 +817,7 @@ void CompareNeighborListFloatArraysWithNans(const DataStructure& dataStructure, } if(exemplaryVal != computedVal) { - float diff = std::fabs(static_cast(exemplaryVal - computedVal)); + float32 diff = std::fabs(static_cast(exemplaryVal - computedVal)); INFO(fmt::format("Bad Neighborlist Comparison\n Exemplary NeighborList:'{}' size:{}\n Computed NeighborList: '{}' size:{} ", exemplaryDataPath.toString(), exemplary.size(), computedPath.toString(), computed.size())); INFO(fmt::format(" NeighborList {}, Index {} Exemplary Value: {} Computed Value: {}", i, j, exemplaryVal, computedVal)) @@ -785,7 +867,7 @@ void CompareNeighborLists(const INeighborList* exemplaryData, const INeighborLis auto computedVal = computed.at(j); if(exemplaryVal != computedVal) { - float diff = std::fabs(static_cast(exemplaryVal - computedVal)); + float32 diff = std::fabs(static_cast(exemplaryVal - computedVal)); INFO(fmt::format("Bad Neighborlist Comparison\n Exemplary NeighborList:'{}' size:{}\n Computed NeighborList: '{}' size:{} ", exemplaryList.getDataPaths()[0].toString(), exemplary.size(), computedList.getDataPaths()[0].toString(), computed.size())); INFO(fmt::format(" NeighborList {}, Index {} Exemplary Value: {} Computed Value: {}", i, j, exemplaryVal, computedVal)) @@ -834,7 +916,7 @@ void CompareNeighborLists(const DataStructure& dataStructure, const DataPath& ex auto computedVal = computed.at(j); if(exemplaryVal != computedVal) { - float diff = std::fabs(static_cast(exemplaryVal - computedVal)); + float32 diff = std::fabs(static_cast(exemplaryVal - computedVal)); INFO(fmt::format(" NeighborList {}, Index {} Exemplary Value: {} Computed Value: {}", i, j, exemplaryVal, computedVal)); REQUIRE(diff < EPSILON); @@ -933,7 +1015,7 @@ void CompareDynamicListArrays(const DataStructure& dataStructure, const DataPath T newNumCells = newEltList.numCells; if(oldNumCells != newNumCells) { - float diff = std::fabs(static_cast(oldNumCells - newNumCells)); + float32 diff = std::fabs(static_cast(oldNumCells - newNumCells)); REQUIRE(diff < EPSILON); } for(T j = 0; j < oldNumCells; ++j) @@ -942,7 +1024,7 @@ void CompareDynamicListArrays(const DataStructure& dataStructure, const DataPath auto newVal = newEltList.cells[j]; if(oldVal != newVal) { - float diff = std::fabs(static_cast(oldVal - newVal)); + float32 diff = std::fabs(static_cast(oldVal - newVal)); REQUIRE(diff < EPSILON); } } @@ -1257,7 +1339,7 @@ inline void CompareDataStructures(const DataStructure& dataStructureA, const Dat /** * @brief Creates a DataArray backed by a DataStore (in memory). - * @tparam T The primitive type to use, i.e. int8, float, double + * @tparam T The primitive type to use, i.e. int8, float32, double * @param dataStructure The DataStructure to use * @param name The name of the DataArray * @param tupleShape The tuple dimensions of the data. If you want to mimic an image then your shape should be {height, width} slowest to fastest dimension @@ -1307,7 +1389,7 @@ inline DataStructure CreateDataStructure() usize numComponents = 1; ShapeType tupleShape = {imageGeomDims[2], imageGeomDims[1], imageGeomDims[0]}; - Float32Array* ci_data = CreateTestDataArray(dataStructure, Constants::k_ConfidenceIndex, tupleShape, {numComponents}, scanData->getId()); + Float32Array* ci_data = CreateTestDataArray(dataStructure, Constants::k_ConfidenceIndex, tupleShape, {numComponents}, scanData->getId()); Int32Array* feature_ids_data = CreateTestDataArray(dataStructure, Constants::k_FeatureIds, tupleShape, {numComponents}, scanData->getId()); Int32Array* phases_data = CreateTestDataArray(dataStructure, "Phases", tupleShape, {numComponents}, scanData->getId()); UInt64Array* voxelIndices = CreateTestDataArray(dataStructure, "Voxel Indices", tupleShape, {numComponents}, scanData->getId()); @@ -1317,7 +1399,7 @@ inline DataStructure CreateDataStructure() numComponents = 3; UInt8Array* ipf_color_data = CreateTestDataArray(dataStructure, "IPF Colors", tupleShape, {numComponents}, scanData->getId()); - Float32Array* euler_data = CreateTestDataArray(dataStructure, "Euler", tupleShape, {numComponents}, scanData->getId()); + Float32Array* euler_data = CreateTestDataArray(dataStructure, "Euler", tupleShape, {numComponents}, scanData->getId()); // Add in another group that holds the phase data such as Laue Class, Lattice Constants, etc. DataGroup* ensembleGroup = DataGroup::Create(dataStructure, "Phase Data", topLevelGroup->getId()); @@ -1599,7 +1681,7 @@ inline void CompareExemplarToGeneratedData(const DataStructure& dataStructure, c } } -inline void CompareAsciiFiles(std::ifstream& computedFile, std::ifstream& exemplarFile, const std::vector& lineIndicesToSkip) +inline void CompareAsciiFiles(std::ifstream& computedFile, std::ifstream& exemplarFile, const std::vector& lineIndicesToSkip) { std::vector computedLines; std::vector exemplarLines; @@ -1613,7 +1695,7 @@ inline void CompareAsciiFiles(std::ifstream& computedFile, std::ifstream& exempl } REQUIRE(computedLines.size() == exemplarLines.size()); - for(size_t i = 0; i < computedLines.size(); ++i) + for(usize i = 0; i < computedLines.size(); ++i) { if(std::find(begin(lineIndicesToSkip), end(lineIndicesToSkip), i) != std::end(lineIndicesToSkip)) { From 6cd0bb1f0b50158876a13be9d33affd69845b1b8 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 14 Apr 2026 09:10:05 -0400 Subject: [PATCH 12/13] FIX: Resolve OOC store format in tests and fix ComputeAvgOrientations bugs Add CreateResolvedDataStore utility that runs the IOCollection format resolver before creating a DataStore, matching the path filter actions use. Update test builder functions to call it so that test-constructed arrays become OOC stores when the OOC plugin is active. Fix three bugs in the OOC ComputeAvgOrientations Rodrigues average: - Allow featureId 0 in accumulation (matching architecture branch) - Start normalization loop from featureId 0 - Add missing continue for zero-count features to avoid divide-by-zero Fix stale GetIOCollection API call in UnitTestCommon (shared_ptr to ref). Signed-off-by: Joey Kleingers --- .../Algorithms/ComputeAvgOrientations.cpp | 5 +-- .../BadDataNeighborOrientationCheckTest.cpp | 8 ++--- .../test/CAxisSegmentFeaturesTest.cpp | 8 +++-- .../test/EBSDSegmentFeaturesFilterTest.cpp | 8 +++-- .../test/ComputeBoundaryCellsTest.cpp | 3 +- .../test/ComputeSurfaceAreaToVolumeTest.cpp | 6 ++-- .../test/ComputeSurfaceFeaturesTest.cpp | 3 +- .../SimplnxCore/test/FillBadDataTest.cpp | 9 +++-- .../SimplnxCore/test/IdentifySampleTest.cpp | 7 ++-- .../test/ScalarSegmentFeaturesTest.cpp | 4 ++- src/simplnx/Utilities/DataStoreUtilities.hpp | 33 +++++++++++++++++++ .../UnitTest/SegmentFeaturesTestUtils.hpp | 16 ++++++--- .../simplnx/UnitTest/UnitTestCommon.hpp | 4 +-- 13 files changed, 84 insertions(+), 30 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp index bdd1998c7d..98641b15a3 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp @@ -421,7 +421,7 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() { const int32 currentFeatureId = featureIdBuf[i]; const int32 currentPhase = phasesBuf[i]; - if(currentFeatureId > 0 && currentPhase > 0) + if(currentPhase > 0) { const uint32 xtal = crystalStructures[currentPhase]; counts[currentFeatureId] += 1.0f; @@ -453,7 +453,7 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() // Second pass: normalize and convert to Euler angles (feature-level only) std::vector localAvgEuler(totalFeatures * 3, 0.0f); - for(usize featureId = 1; featureId < totalFeatures; featureId++) + for(usize featureId = 0; featureId < totalFeatures; featureId++) { if(m_ShouldCancel) { @@ -467,6 +467,7 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() localAvgQuats[fi + 1] = identityQuat.y(); localAvgQuats[fi + 2] = identityQuat.z(); localAvgQuats[fi + 3] = identityQuat.w(); + continue; } ebsdlib::QuatF curAvgQuat(localAvgQuats[fi], localAvgQuats[fi + 1], localAvgQuats[fi + 2], localAvgQuats[fi + 3]); diff --git a/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp b/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp index f4fc6a0a51..056a7c317c 100644 --- a/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/BadDataNeighborOrientationCheckTest.cpp @@ -59,14 +59,14 @@ void BuildOrientationOctantDataset(DataStructure& ds) AttributeMatrix* ensembleAM = AttributeMatrix::Create(ds, Constants::k_Cell_Ensemble_Data, {2}, imageGeom->getId()); // Crystal Structures: [999 (Unknown), 1 (Cubic_High)] - auto csStore = DataStoreUtilities::CreateDataStore({2}, {1}, IDataAction::Mode::Execute); + auto csStore = DataStoreUtilities::CreateResolvedDataStore(ds, VerificationConstants::k_CStuctsArrayPath, {2}, {1}); auto* crystalStructures = DataArray::Create(ds, VerificationConstants::k_CStuctsName, csStore, ensembleAM->getId()); auto& csRef = crystalStructures->getDataStoreRef(); csRef.setValue(0, 999); csRef.setValue(1, 1); // Phases: all phase 1 — bulk-write per Z-slice to avoid per-element OOC overhead - auto phasesStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto phasesStore = DataStoreUtilities::CreateResolvedDataStore(ds, VerificationConstants::k_PhasesArrayPath, cellTupleShape, {1}); auto* phases = DataArray::Create(ds, VerificationConstants::k_PhasesName, phasesStore, cellAM->getId()); auto& phasesRef = phases->getDataStoreRef(); { @@ -79,7 +79,7 @@ void BuildOrientationOctantDataset(DataStructure& ds) } // Mask: ~40% bad voxels (mask=0) — bulk-write per Z-slice - auto maskStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + auto maskStore = DataStoreUtilities::CreateResolvedDataStore(ds, VerificationConstants::k_MaskArrayPath, cellTupleShape, {1}); auto* mask = DataArray::Create(ds, VerificationConstants::k_MaskName, maskStore, cellAM->getId()); auto& maskRef = mask->getDataStoreRef(); { @@ -101,7 +101,7 @@ void BuildOrientationOctantDataset(DataStructure& ds) } // Quats: 8 octants with distinct base quaternions (15 deg apart about Z axis) — bulk-write per Z-slice - auto quatsStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {4}, IDataAction::Mode::Execute); + auto quatsStore = DataStoreUtilities::CreateResolvedDataStore(ds, VerificationConstants::k_QuatsArrayPath, cellTupleShape, {4}); auto* quats = DataArray::Create(ds, VerificationConstants::k_QuatsName, quatsStore, cellAM->getId()); auto& quatsRef = quats->getDataStoreRef(); { diff --git a/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp b/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp index d806804c4b..7f6c204ad9 100644 --- a/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/CAxisSegmentFeaturesTest.cpp @@ -9,6 +9,7 @@ #include "simplnx/UnitTest/SegmentFeaturesTestUtils.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" #include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -286,7 +287,7 @@ TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: FaceEdgeVertex Connectivit // Quaternions: background = 60° X-rotation, pairs = identity and 30° (EBSDlib order: x,y,z,w) const float32 bgHalf = 60.0f * k_DegToRad * 0.5f; - auto quatsDS = DataStoreUtilities::CreateDataStore(cellShape, {4}, IDataAction::Mode::Execute); + auto quatsDS = DataStoreUtilities::CreateResolvedDataStore(ds, cellDataPath.createChildPath("Quats"), cellShape, {4}); auto* quatsArr = DataArray::Create(ds, "Quats", quatsDS, am.getId()); auto& quatsStore = quatsArr->getDataStoreRef(); for(usize i = 0; i < 27; i++) @@ -312,13 +313,14 @@ TEST_CASE("OrientationAnalysis::CAxisSegmentFeatures: FaceEdgeVertex Connectivit quatsStore[idx * 4 + 3] = std::cos(pairHalf); } - auto phasesDS = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto phasesDS = DataStoreUtilities::CreateResolvedDataStore(ds, cellDataPath.createChildPath("Phases"), cellShape, {1}); auto* phasesArr = DataArray::Create(ds, "Phases", phasesDS, am.getId()); phasesArr->fill(1); const ShapeType ensShape = {2}; auto* ensAM = AttributeMatrix::Create(ds, "CellEnsembleData", ensShape, geom.getId()); - auto crystDS = DataStoreUtilities::CreateDataStore(ensShape, {1}, IDataAction::Mode::Execute); + const DataPath crystStructsPath = geomPath.createChildPath("CellEnsembleData").createChildPath("CrystalStructures"); + auto crystDS = DataStoreUtilities::CreateResolvedDataStore(ds, crystStructsPath, ensShape, {1}); auto* crystArr = DataArray::Create(ds, "CrystalStructures", crystDS, ensAM->getId()); auto& crystStore = crystArr->getDataStoreRef(); crystStore[0] = 999; diff --git a/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp b/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp index bff8e2b2e8..0bc8600574 100644 --- a/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp +++ b/src/Plugins/OrientationAnalysis/test/EBSDSegmentFeaturesFilterTest.cpp @@ -9,6 +9,7 @@ #include "simplnx/UnitTest/SegmentFeaturesTestUtils.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" #include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -292,7 +293,7 @@ TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: FaceEdgeVertex Connectivity // Quaternions: background = 60° X-rotation, pairs = identity (EBSDlib order: x,y,z,w) const float32 bgHalf = 60.0f * k_DegToRad * 0.5f; - auto quatsDS = DataStoreUtilities::CreateDataStore(cellShape, {4}, IDataAction::Mode::Execute); + auto quatsDS = DataStoreUtilities::CreateResolvedDataStore(ds, cellDataPath.createChildPath("Quats"), cellShape, {4}); auto* quatsArr = DataArray::Create(ds, "Quats", quatsDS, am.getId()); auto& quatsStore = quatsArr->getDataStoreRef(); for(usize i = 0; i < 27; i++) @@ -321,14 +322,15 @@ TEST_CASE("OrientationAnalysis::EBSDSegmentFeatures: FaceEdgeVertex Connectivity } // Phases: all phase 1 - auto phasesDS = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + auto phasesDS = DataStoreUtilities::CreateResolvedDataStore(ds, cellDataPath.createChildPath("Phases"), cellShape, {1}); auto* phasesArr = DataArray::Create(ds, "Phases", phasesDS, am.getId()); phasesArr->fill(1); // CrystalStructures: phase 0 = unknown, phase 1 = Cubic_High const ShapeType ensShape = {2}; auto* ensAM = AttributeMatrix::Create(ds, "CellEnsembleData", ensShape, geom.getId()); - auto crystDS = DataStoreUtilities::CreateDataStore(ensShape, {1}, IDataAction::Mode::Execute); + const DataPath crystStructsPath = geomPath.createChildPath("CellEnsembleData").createChildPath("CrystalStructures"); + auto crystDS = DataStoreUtilities::CreateResolvedDataStore(ds, crystStructsPath, ensShape, {1}); auto* crystArr = DataArray::Create(ds, "CrystalStructures", crystDS, ensAM->getId()); auto& crystStore = crystArr->getDataStoreRef(); crystStore[0] = 999; diff --git a/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp b/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp index 76f665f675..8899a46f49 100644 --- a/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeBoundaryCellsTest.cpp @@ -39,7 +39,8 @@ void BuildOctantFeatureIds(DataStructure& ds) auto* cellAM = AttributeMatrix::Create(ds, Constants::k_CellData, cellTupleShape, imageGeom->getId()); imageGeom->setCellData(*cellAM); - auto store = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + const DataPath featureIdsPath = DataPath({k_GeomName, Constants::k_CellData, Constants::k_FeatureIds}); + auto store = DataStoreUtilities::CreateResolvedDataStore(ds, featureIdsPath, cellTupleShape, {1}); auto* featureIds = DataArray::Create(ds, Constants::k_FeatureIds, store, cellAM->getId()); auto& fidsRef = featureIds->getDataStoreRef(); const usize sliceSize = k_DimY * k_DimX; diff --git a/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp b/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp index 9026379e54..0cbf517bd5 100644 --- a/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeSurfaceAreaToVolumeTest.cpp @@ -44,7 +44,8 @@ void BuildOctantWithNumCells(DataStructure& ds) auto* cellAM = AttributeMatrix::Create(ds, Constants::k_CellData, cellTupleShape, imageGeom->getId()); imageGeom->setCellData(*cellAM); - auto fidsStore = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + const DataPath featureIdsPath = DataPath({k_GeomName, Constants::k_CellData, Constants::k_FeatureIds}); + auto fidsStore = DataStoreUtilities::CreateResolvedDataStore(ds, featureIdsPath, cellTupleShape, {1}); auto* featureIds = DataArray::Create(ds, Constants::k_FeatureIds, fidsStore, cellAM->getId()); auto& fidsRef = featureIds->getDataStoreRef(); @@ -74,7 +75,8 @@ void BuildOctantWithNumCells(DataStructure& ds) auto* cellFeatureAM = AttributeMatrix::Create(ds, Constants::k_CellFeatureData, {static_cast(k_NumOctantFeatures + 1)}, imageGeom->getId()); // NumCells is small (9 tuples) — per-element writes are fine - auto numCellsStore = DataStoreUtilities::CreateDataStore({static_cast(k_NumOctantFeatures + 1)}, {1}, IDataAction::Mode::Execute); + const DataPath numCellsPath = DataPath({k_GeomName, Constants::k_CellFeatureData, Constants::k_NumElements}); + auto numCellsStore = DataStoreUtilities::CreateResolvedDataStore(ds, numCellsPath, {static_cast(k_NumOctantFeatures + 1)}, {1}); auto* numCells = DataArray::Create(ds, Constants::k_NumElements, numCellsStore, cellFeatureAM->getId()); auto& numCellsRef = numCells->getDataStoreRef(); std::vector localNumCells(featureCounts.begin(), featureCounts.end()); diff --git a/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp b/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp index aa0c906116..24f3c7d9f4 100644 --- a/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp +++ b/src/Plugins/SimplnxCore/test/ComputeSurfaceFeaturesTest.cpp @@ -145,7 +145,8 @@ void BuildBlockFeatureIds(DataStructure& ds) auto* cellAM = AttributeMatrix::Create(ds, Constants::k_CellData, cellTupleShape, imageGeom->getId()); imageGeom->setCellData(*cellAM); - auto store = DataStoreUtilities::CreateDataStore(cellTupleShape, {1}, IDataAction::Mode::Execute); + const DataPath featureIdsPath = DataPath({k_BenchGeomName, Constants::k_CellData, Constants::k_FeatureIds}); + auto store = DataStoreUtilities::CreateResolvedDataStore(ds, featureIdsPath, cellTupleShape, {1}); auto* featureIds = DataArray::Create(ds, Constants::k_FeatureIds, store, cellAM->getId()); auto& fidsRef = featureIds->getDataStoreRef(); const usize sliceSize = k_BenchDim * k_BenchDim; diff --git a/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp b/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp index cb1b79b284..d7d5b0cf2a 100644 --- a/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp +++ b/src/Plugins/SimplnxCore/test/FillBadDataTest.cpp @@ -36,11 +36,13 @@ void BuildFillBadDataTestData(DataStructure& ds, usize dimX, usize dimY, usize d auto* cellAM = AttributeMatrix::Create(ds, "CellData", cellShape, imageGeom->getId()); imageGeom->setCellData(*cellAM); - auto featureIdsDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath featureIdsPath = DataPath({"DataContainer", "CellData", "FeatureIds"}); + auto featureIdsDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, featureIdsPath, cellShape, {1}); auto* featureIdsArray = DataArray::Create(ds, "FeatureIds", featureIdsDataStore, cellAM->getId()); auto& featureIdsStore = featureIdsArray->getDataStoreRef(); - auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath phasesPath = DataPath({"DataContainer", "CellData", "Phases"}); + auto phasesDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, phasesPath, cellShape, {1}); auto* phasesArray = DataArray::Create(ds, "Phases", phasesDataStore, cellAM->getId()); auto& phasesStore = phasesArray->getDataStoreRef(); @@ -523,7 +525,8 @@ TEST_CASE("SimplnxCore::FillBadData: 200x200x200 Ignored Arrays", "[Core][FillBa // Add an extra "IgnoredArray" filled with a sentinel value auto& cellAM = dataStructure.getDataRefAs(DataPath({"DataContainer", "CellData"})); - auto ignoredDataStore = DataStoreUtilities::CreateDataStore(cellAM.getShape(), {1}, IDataAction::Mode::Execute); + const DataPath ignoredArrayPath = DataPath({"DataContainer", "CellData", "IgnoredArray"}); + auto ignoredDataStore = DataStoreUtilities::CreateResolvedDataStore(dataStructure, ignoredArrayPath, cellAM.getShape(), {1}); auto* ignoredArray = DataArray::Create(dataStructure, "IgnoredArray", ignoredDataStore, cellAM.getId()); auto& ignoredStore = ignoredArray->getDataStoreRef(); ignoredStore.fill(k_Sentinel); diff --git a/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp b/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp index 81c00298f7..32b862e321 100644 --- a/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp +++ b/src/Plugins/SimplnxCore/test/IdentifySampleTest.cpp @@ -52,7 +52,8 @@ void BuildIdentifySampleTestData(DataStructure& ds, usize dimX, usize dimY, usiz auto* cellAM = AttributeMatrix::Create(ds, "CellData", cellShape, imageGeom->getId()); imageGeom->setCellData(*cellAM); - auto maskDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath maskPath = DataPath({geomName, "CellData", "Mask"}); + auto maskDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, maskPath, cellShape, {1}); auto* maskArray = DataArray::Create(ds, "Mask", maskDataStore, cellAM->getId()); auto& maskStore = maskArray->getDataStoreRef(); @@ -89,13 +90,13 @@ void BuildIdentifySampleTestData(DataStructure& ds, usize dimX, usize dimY, usiz if(good) { const float32 h1 = std::sqrt((static_cast(x) - h1cx) * (static_cast(x) - h1cx) + (static_cast(y) - h1cy) * (static_cast(y) - h1cy) + - (static_cast(z) - h1cz) * (static_cast(z) - h1cz)); + (static_cast(z) - h1cz) * (static_cast(z) - h1cz)); if(h1 < h1r) { good = false; } const float32 h2 = std::sqrt((static_cast(x) - h2cx) * (static_cast(x) - h2cx) + (static_cast(y) - h2cy) * (static_cast(y) - h2cy) + - (static_cast(z) - h2cz) * (static_cast(z) - h2cz)); + (static_cast(z) - h2cz) * (static_cast(z) - h2cz)); if(h2 < h2r) { good = false; diff --git a/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp b/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp index 4ef3a6fb0d..b569240f20 100644 --- a/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp +++ b/src/Plugins/SimplnxCore/test/ScalarSegmentFeaturesTest.cpp @@ -7,6 +7,7 @@ #include "simplnx/UnitTest/SegmentFeaturesTestUtils.hpp" #include "simplnx/UnitTest/UnitTestCommon.hpp" #include "simplnx/Utilities/AlgorithmDispatch.hpp" +#include "simplnx/Utilities/DataStoreUtilities.hpp" #include @@ -127,7 +128,8 @@ TEST_CASE("SimplnxCore::ScalarSegmentFeatures: FaceEdgeVertex Connectivity", "[S auto setupScalar = [](Arguments& args, DataStructure& ds, const DataPath& geomPath, const DataPath& cellDataPath, ChoicesParameter::ValueType neighborScheme) { const ShapeType cellShape = {3, 3, 3}; auto& am = ds.getDataRefAs(cellDataPath); - auto scalarDS = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath scalarPath = cellDataPath.createChildPath("ScalarData"); + auto scalarDS = DataStoreUtilities::CreateResolvedDataStore(ds, scalarPath, cellShape, {1}); auto* scalarArr = DataArray::Create(ds, "ScalarData", scalarDS, am.getId()); auto& store = scalarArr->getDataStoreRef(); store.fill(0); diff --git a/src/simplnx/Utilities/DataStoreUtilities.hpp b/src/simplnx/Utilities/DataStoreUtilities.hpp index 5a4014c4a8..e6677539cc 100644 --- a/src/simplnx/Utilities/DataStoreUtilities.hpp +++ b/src/simplnx/Utilities/DataStoreUtilities.hpp @@ -72,6 +72,39 @@ std::shared_ptr> CreateDataStore(const ShapeType& tupleShap } } +/** + * @brief Creates a DataStore whose format is resolved through the IOCollection's + * format resolver, exactly like filter CreateArrayActions do. + * + * Use this instead of CreateDataStore when you need the store format to respect + * user preferences (OOC thresholds, forced OOC mode, etc.). This is the correct + * function for test code that builds DataArrays which should be OOC when the + * OOC plugin is active. + * + * @tparam T Primitive type (int8, float32, uint64, etc.) + * @param dataStructure The DataStructure the array will live in (needed for format resolution) + * @param arrayPath The DataPath where the array will be inserted + * @param tupleShape The tuple dimensions + * @param componentShape The component dimensions + * @return Shared pointer to the created AbstractDataStore + */ +template +std::shared_ptr> CreateResolvedDataStore(const DataStructure& dataStructure, const DataPath& arrayPath, const ShapeType& tupleShape, const ShapeType& componentShape) +{ + uint64 numElements = 1; + for(auto dim : tupleShape) + { + numElements *= dim; + } + for(auto dim : componentShape) + { + numElements *= dim; + } + uint64 requiredBytes = numElements * sizeof(T); + std::string resolvedFormat = GetIOCollection().resolveFormat(dataStructure, arrayPath, GetDataType(), requiredBytes); + return GetIOCollection().createDataStoreWithType(resolvedFormat, tupleShape, componentShape); +} + /** * @brief Simple factory that creates a ListStore with the given properties. * diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp b/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp index 1649797a30..5bf77e811c 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/SegmentFeaturesTestUtils.hpp @@ -56,7 +56,8 @@ inline void BuildScalarTestData(DataStructure& ds, const ShapeType& cellShape, D const usize dimY = cellShape[1]; const usize dimX = cellShape[2]; - auto scalarDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath scalarPath = ds.getDataPathsForId(amId)[0].createChildPath(arrayName); + auto scalarDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, scalarPath, cellShape, {1}); auto* scalarArray = DataArray::Create(ds, arrayName, scalarDataStore, amId); auto& store = scalarArray->getDataStoreRef(); @@ -119,11 +120,14 @@ inline void BuildOrientationTestData(DataStructure& ds, const ShapeType& cellSha const usize dimY = cellShape[1]; const usize dimX = cellShape[2]; - auto quatsDataStore = DataStoreUtilities::CreateDataStore(cellShape, {4}, IDataAction::Mode::Execute); + const DataPath amPath = ds.getDataPathsForId(amId)[0]; + const DataPath quatsPath = amPath.createChildPath("Quats"); + auto quatsDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, quatsPath, cellShape, {4}); auto* quatsArray = DataArray::Create(ds, "Quats", quatsDataStore, amId); auto& quatsStore = quatsArray->getDataStoreRef(); - auto phasesDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath phasesPath = amPath.createChildPath("Phases"); + auto phasesDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, phasesPath, cellShape, {1}); auto* phasesArray = DataArray::Create(ds, "Phases", phasesDataStore, amId); auto& phasesStore = phasesArray->getDataStoreRef(); @@ -223,7 +227,8 @@ inline void BuildOrientationTestData(DataStructure& ds, const ShapeType& cellSha // Create CellEnsembleData with CrystalStructures const ShapeType ensembleTupleShape = {2}; auto* ensembleAM = AttributeMatrix::Create(ds, "CellEnsembleData", ensembleTupleShape, geomId); - auto crystalDataStore = DataStoreUtilities::CreateDataStore(ensembleTupleShape, {1}, IDataAction::Mode::Execute); + const DataPath crystalStructsPath = ds.getDataPathsForId(geomId)[0].createChildPath("CellEnsembleData").createChildPath("CrystalStructures"); + auto crystalDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, crystalStructsPath, ensembleTupleShape, {1}); auto* crystalStructsArray = DataArray::Create(ds, "CrystalStructures", crystalDataStore, ensembleAM->getId()); auto& crystalStructsStore = crystalStructsArray->getDataStoreRef(); crystalStructsStore[0] = 999; // Phase 0: Unknown @@ -248,7 +253,8 @@ inline void BuildSphericalMask(DataStructure& ds, const ShapeType& cellShape, Da const usize dimY = cellShape[1]; const usize dimX = cellShape[2]; - auto maskDataStore = DataStoreUtilities::CreateDataStore(cellShape, {1}, IDataAction::Mode::Execute); + const DataPath maskPath = ds.getDataPathsForId(amId)[0].createChildPath(maskName); + auto maskDataStore = DataStoreUtilities::CreateResolvedDataStore(ds, maskPath, cellShape, {1}); auto* maskArray = DataArray::Create(ds, maskName, maskDataStore, amId); auto& maskStore = maskArray->getDataStoreRef(); diff --git a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp index c1ee31e7a3..864aa19ca1 100644 --- a/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp +++ b/test/UnitTestCommon/include/simplnx/UnitTest/UnitTestCommon.hpp @@ -320,8 +320,8 @@ inline IDataStore::StoreType ExpectedStoreType() auto* prefs = Application::GetOrCreateInstance()->getPreferences(); if(prefs->forceOocData()) { - auto ioCollection = DataStoreUtilities::GetIOCollection(); - auto manager = ioCollection->getManager(prefs->largeDataFormat()); + auto& ioCollection = DataStoreUtilities::GetIOCollection(); + auto manager = ioCollection.getManager(prefs->largeDataFormat()); if(manager != nullptr) { return IDataStore::StoreType::OutOfCore; From 1579e6441c06e4e6d601942968e766f657f86c76 Mon Sep 17 00:00:00 2001 From: Joey Kleingers Date: Tue, 14 Apr 2026 10:41:39 -0400 Subject: [PATCH 13/13] DOCS: Add comprehensive Doxygen and inline documentation for OOC-optimized algorithms Adds extensive documentation across all out-of-core optimized filter algorithms explaining what each algorithm does and why the OOC variant works the way it does. Targets readers with no prior OOC knowledge. - Headers: Doxygen @class, @brief, @param on all classes, methods, InputValues structs, and member variables - Source files: file-level overviews, Doxygen on operator()(), and inline comments explaining rolling windows, buffer strategies, dispatch logic, and OOC rationale - Filter docs: Algorithm sections with In-Core/Out-of-Core/Performance subsections added to ~45 filter markdown files - Key utilities: SliceBufferedTransfer.hpp and TupleTransfer.hpp documented as core OOC infrastructure --- .../docs/AlignSectionsMisorientationFilter.md | 14 + .../AlignSectionsMutualInformationFilter.md | 14 + .../BadDataNeighborOrientationCheckFilter.md | 26 ++ .../docs/CAxisSegmentFeaturesFilter.md | 16 + .../docs/ComputeAvgCAxesFilter.md | 16 + .../docs/ComputeAvgOrientationsFilter.md | 20 ++ .../docs/ComputeCAxisLocationsFilter.md | 16 + ...FeatureNeighborCAxisMisalignmentsFilter.md | 16 + ...tureReferenceCAxisMisorientationsFilter.md | 18 ++ ...teFeatureReferenceMisorientationsFilter.md | 22 ++ .../docs/ComputeGBCDFilter.md | 18 ++ .../docs/ComputeGBCDMetricBasedFilter.md | 18 ++ .../docs/ComputeGBCDPoleFigureFilter.md | 26 ++ .../docs/ComputeIPFColorsFilter.md | 24 ++ .../ComputeKernelAvgMisorientationsFilter.md | 16 + .../docs/ComputeTwinBoundariesFilter.md | 20 ++ .../docs/ConvertOrientationsFilter.md | 16 + .../docs/EBSDSegmentFeaturesFilter.md | 16 + .../docs/MergeTwinsFilter.md | 16 + .../NeighborOrientationCorrelationFilter.md | 14 + .../docs/RotateEulerRefFrameFilter.md | 16 + .../AlignSectionsMisorientation.cpp | 14 + .../AlignSectionsMisorientation.hpp | 99 ++++-- .../AlignSectionsMutualInformation.hpp | 120 +++++-- .../BadDataNeighborOrientationCheck.cpp | 16 + .../BadDataNeighborOrientationCheck.hpp | 65 +++- ...adDataNeighborOrientationCheckScanline.cpp | 109 +++++-- ...adDataNeighborOrientationCheckScanline.hpp | 56 +++- ...adDataNeighborOrientationCheckWorklist.cpp | 66 +++- ...adDataNeighborOrientationCheckWorklist.hpp | 54 +++- .../Algorithms/CAxisSegmentFeatures.hpp | 158 ++++++++-- .../Filters/Algorithms/ComputeAvgCAxes.cpp | 33 +- .../Filters/Algorithms/ComputeAvgCAxes.hpp | 42 ++- .../Algorithms/ComputeAvgOrientations.cpp | 39 ++- .../Algorithms/ComputeAvgOrientations.hpp | 103 ++++-- .../Algorithms/ComputeCAxisLocations.cpp | 17 +- .../Algorithms/ComputeCAxisLocations.hpp | 35 ++- ...mputeFeatureNeighborCAxisMisalignments.cpp | 25 +- ...mputeFeatureNeighborCAxisMisalignments.hpp | 40 ++- ...teFeatureReferenceCAxisMisorientations.cpp | 33 +- ...teFeatureReferenceCAxisMisorientations.hpp | 49 ++- ...ComputeFeatureReferenceMisorientations.cpp | 31 +- ...ComputeFeatureReferenceMisorientations.hpp | 48 ++- .../Filters/Algorithms/ComputeGBCD.cpp | 46 ++- .../Filters/Algorithms/ComputeGBCD.hpp | 51 ++- .../Algorithms/ComputeGBCDMetricBased.cpp | 40 ++- .../ComputeGBCDPoleFigureDirect.cpp | 145 ++++++--- .../ComputeGBCDPoleFigureDirect.hpp | 72 ++++- .../ComputeGBCDPoleFigureScanline.cpp | 84 +++-- .../ComputeGBCDPoleFigureScanline.hpp | 49 ++- .../Filters/Algorithms/ComputeIPFColors.cpp | 12 + .../Filters/Algorithms/ComputeIPFColors.hpp | 58 +++- .../Algorithms/ComputeIPFColorsDirect.cpp | 98 +++++- .../Algorithms/ComputeIPFColorsDirect.hpp | 59 +++- .../Algorithms/ComputeIPFColorsScanline.cpp | 70 ++++- .../Algorithms/ComputeIPFColorsScanline.hpp | 50 ++- .../ComputeKernelAvgMisorientations.cpp | 35 ++- .../ComputeKernelAvgMisorientations.hpp | 53 +++- .../Algorithms/ComputeTwinBoundaries.cpp | 43 ++- .../Algorithms/ComputeTwinBoundaries.hpp | 55 +++- .../Algorithms/ConvertOrientations.cpp | 44 ++- .../Algorithms/ConvertOrientations.hpp | 33 +- .../Algorithms/EBSDSegmentFeatures.hpp | 179 ++++++++--- .../Filters/Algorithms/MergeTwins.cpp | 25 +- .../Filters/Algorithms/MergeTwins.hpp | 56 +++- .../Filters/Algorithms/ReadAngData.cpp | 48 ++- .../Filters/Algorithms/ReadAngData.hpp | 45 ++- .../Filters/Algorithms/ReadCtfData.cpp | 26 +- .../Filters/Algorithms/ReadCtfData.hpp | 12 +- .../Filters/Algorithms/ReadH5Ebsd.cpp | 29 +- .../Filters/Algorithms/ReadH5Ebsd.hpp | 12 +- .../Filters/Algorithms/ReadH5EspritData.cpp | 25 +- .../Filters/Algorithms/ReadH5EspritData.hpp | 19 +- .../Algorithms/RotateEulerRefFrame.cpp | 16 +- .../Algorithms/RotateEulerRefFrame.hpp | 36 ++- .../Filters/Algorithms/WriteGBCDGMTFile.cpp | 15 + .../Filters/Algorithms/WriteGBCDGMTFile.hpp | 24 +- .../Algorithms/WriteGBCDTriangleData.cpp | 34 +- .../Algorithms/WriteGBCDTriangleData.hpp | 22 +- .../Filters/Algorithms/WritePoleFigure.hpp | 11 +- .../AlignSectionsFeatureCentroidFilter.md | 13 + .../docs/ComputeArrayStatisticsFilter.md | 4 + .../docs/ComputeBoundaryCellsFilter.md | 29 ++ .../docs/ComputeEuclideanDistMapFilter.md | 6 + .../docs/ComputeFeatureCentroidsFilter.md | 10 + .../docs/ComputeFeatureClusteringFilter.md | 4 + .../docs/ComputeFeatureNeighborsFilter.md | 31 ++ .../docs/ComputeFeatureSizesFilter.md | 10 + .../SimplnxCore/docs/ComputeKMedoidsFilter.md | 29 ++ .../docs/ComputeSurfaceAreaToVolumeFilter.md | 32 +- .../docs/ComputeSurfaceFeaturesFilter.md | 30 ++ .../docs/CropImageGeometryFilter.md | 8 + src/Plugins/SimplnxCore/docs/DBSCANFilter.md | 27 ++ .../docs/ErodeDilateBadDataFilter.md | 27 ++ .../ErodeDilateCoordinationNumberFilter.md | 26 ++ .../SimplnxCore/docs/ErodeDilateMaskFilter.md | 24 ++ .../SimplnxCore/docs/IdentifySampleFilter.md | 34 ++ .../docs/MultiThresholdObjectsFilter.md | 25 ++ .../docs/QuickSurfaceMeshFilter.md | 34 ++ .../RegularGridSampleSurfaceMeshFilter.md | 11 +- ...ementAttributesWithNeighborValuesFilter.md | 26 ++ .../docs/RequireMinimumSizeFeaturesFilter.md | 14 + .../docs/ScalarSegmentFeaturesFilter.md | 15 + .../SimplnxCore/docs/SurfaceNetsFilter.md | 40 +++ .../AlignSectionsFeatureCentroid.cpp | 32 +- .../AlignSectionsFeatureCentroid.hpp | 72 +++-- .../Algorithms/ComputeArrayStatistics.cpp | 8 + .../Algorithms/ComputeArrayStatistics.hpp | 11 +- .../Algorithms/ComputeBoundaryCells.cpp | 27 ++ .../Algorithms/ComputeBoundaryCells.hpp | 62 +++- .../Algorithms/ComputeBoundaryCellsDirect.cpp | 62 +++- .../Algorithms/ComputeBoundaryCellsDirect.hpp | 48 ++- .../ComputeBoundaryCellsScanline.cpp | 119 ++++++- .../ComputeBoundaryCellsScanline.hpp | 62 +++- .../Algorithms/ComputeEuclideanDistMap.cpp | 25 +- .../Algorithms/ComputeEuclideanDistMap.hpp | 74 ++++- .../Algorithms/ComputeFeatureCentroids.cpp | 36 ++- .../Algorithms/ComputeFeatureCentroids.hpp | 48 ++- .../Algorithms/ComputeFeatureClustering.cpp | 19 +- .../Algorithms/ComputeFeatureClustering.hpp | 63 ++-- .../Algorithms/ComputeFeatureNeighbors.cpp | 23 ++ .../Algorithms/ComputeFeatureNeighbors.hpp | 74 ++++- .../ComputeFeatureNeighborsDirect.cpp | 43 ++- .../ComputeFeatureNeighborsDirect.hpp | 27 +- .../ComputeFeatureNeighborsScanline.cpp | 54 +++- .../ComputeFeatureNeighborsScanline.hpp | 29 +- .../Algorithms/ComputeFeatureSizes.cpp | 10 +- .../Algorithms/ComputeFeatureSizes.hpp | 50 +-- .../Filters/Algorithms/ComputeKMedoids.cpp | 23 ++ .../Filters/Algorithms/ComputeKMedoids.hpp | 79 ++++- .../Algorithms/ComputeKMedoidsDirect.cpp | 60 ++++ .../Algorithms/ComputeKMedoidsDirect.hpp | 45 ++- .../Algorithms/ComputeKMedoidsScanline.cpp | 90 +++++- .../Algorithms/ComputeKMedoidsScanline.hpp | 55 +++- .../Algorithms/ComputeSurfaceAreaToVolume.cpp | 28 ++ .../Algorithms/ComputeSurfaceAreaToVolume.hpp | 65 +++- .../ComputeSurfaceAreaToVolumeDirect.cpp | 92 +++++- .../ComputeSurfaceAreaToVolumeDirect.hpp | 56 +++- .../ComputeSurfaceAreaToVolumeScanline.cpp | 134 ++++++-- .../ComputeSurfaceAreaToVolumeScanline.hpp | 60 +++- .../Algorithms/ComputeSurfaceFeatures.cpp | 25 ++ .../Algorithms/ComputeSurfaceFeatures.hpp | 64 +++- .../ComputeSurfaceFeaturesDirect.cpp | 127 +++++++- .../ComputeSurfaceFeaturesDirect.hpp | 53 +++- .../ComputeSurfaceFeaturesScanline.cpp | 213 ++++++++++--- .../ComputeSurfaceFeaturesScanline.hpp | 61 +++- .../Filters/Algorithms/CropImageGeometry.cpp | 5 +- .../Filters/Algorithms/CropImageGeometry.hpp | 65 ++-- .../SimplnxCore/Filters/Algorithms/DBSCAN.cpp | 23 ++ .../SimplnxCore/Filters/Algorithms/DBSCAN.hpp | 81 ++++- .../Filters/Algorithms/DBSCANDirect.cpp | 40 ++- .../Filters/Algorithms/DBSCANDirect.hpp | 38 ++- .../Filters/Algorithms/DBSCANScanline.cpp | 62 +++- .../Filters/Algorithms/DBSCANScanline.hpp | 47 ++- .../Filters/Algorithms/ErodeDilateBadData.cpp | 121 +++++++- .../Filters/Algorithms/ErodeDilateBadData.hpp | 88 +++++- .../ErodeDilateCoordinationNumber.cpp | 82 ++++- .../ErodeDilateCoordinationNumber.hpp | 77 ++++- .../Filters/Algorithms/ErodeDilateMask.cpp | 93 +++++- .../Filters/Algorithms/ErodeDilateMask.hpp | 78 ++++- .../Filters/Algorithms/FillBadData.cpp | 33 ++ .../Filters/Algorithms/FillBadData.hpp | 53 +++- .../Filters/Algorithms/FillBadDataBFS.cpp | 31 ++ .../Filters/Algorithms/FillBadDataBFS.hpp | 76 ++++- .../Filters/Algorithms/FillBadDataCCL.cpp | 177 ++++++++--- .../Filters/Algorithms/FillBadDataCCL.hpp | 140 ++++++++- .../Filters/Algorithms/IdentifySample.cpp | 33 ++ .../Filters/Algorithms/IdentifySample.hpp | 77 ++++- .../Filters/Algorithms/IdentifySampleBFS.cpp | 22 ++ .../Filters/Algorithms/IdentifySampleBFS.hpp | 71 ++++- .../Filters/Algorithms/IdentifySampleCCL.cpp | 45 +++ .../Filters/Algorithms/IdentifySampleCCL.hpp | 96 +++++- .../Algorithms/IdentifySampleCommon.hpp | 47 ++- .../Algorithms/MultiThresholdObjects.cpp | 24 ++ .../Algorithms/MultiThresholdObjects.hpp | 84 +++-- .../MultiThresholdObjectsDirect.cpp | 30 ++ .../MultiThresholdObjectsDirect.hpp | 35 ++- .../MultiThresholdObjectsScanline.cpp | 71 ++++- .../MultiThresholdObjectsScanline.hpp | 47 ++- .../Filters/Algorithms/QuickSurfaceMesh.cpp | 23 ++ .../Filters/Algorithms/QuickSurfaceMesh.hpp | 64 ++-- .../Algorithms/QuickSurfaceMeshDirect.cpp | 118 +++++++ .../Algorithms/QuickSurfaceMeshDirect.hpp | 70 ++++- .../Algorithms/QuickSurfaceMeshScanline.cpp | 161 +++++++++- .../Algorithms/QuickSurfaceMeshScanline.hpp | 104 ++++++- .../Filters/Algorithms/ReadHDF5Dataset.cpp | 18 ++ .../Filters/Algorithms/ReadHDF5Dataset.hpp | 20 +- .../Filters/Algorithms/ReadStlFile.cpp | 4 + .../Filters/Algorithms/ReadStlFile.hpp | 8 +- ...aceElementAttributesWithNeighborValues.cpp | 148 ++++++++- ...aceElementAttributesWithNeighborValues.hpp | 78 ++++- .../Algorithms/RequireMinimumSizeFeatures.cpp | 21 +- .../Algorithms/RequireMinimumSizeFeatures.hpp | 77 +++-- .../Algorithms/ScalarSegmentFeatures.hpp | 127 ++++++-- .../Filters/Algorithms/SurfaceNets.cpp | 23 ++ .../Filters/Algorithms/SurfaceNets.hpp | 77 +++-- .../Filters/Algorithms/SurfaceNetsDirect.cpp | 104 ++++++- .../Filters/Algorithms/SurfaceNetsDirect.hpp | 57 +++- .../Algorithms/SurfaceNetsScanline.cpp | 180 ++++++++++- .../Algorithms/SurfaceNetsScanline.hpp | 96 +++++- .../Filters/Algorithms/TupleTransfer.hpp | 293 ++++++++++++++---- .../WriteAvizoRectilinearCoordinate.cpp | 21 +- .../WriteAvizoRectilinearCoordinate.hpp | 20 +- .../WriteAvizoUniformCoordinate.cpp | 13 + .../WriteAvizoUniformCoordinate.hpp | 19 +- .../Utilities/SliceBufferedTransfer.hpp | 192 ++++++++++-- 206 files changed, 9057 insertions(+), 1449 deletions(-) diff --git a/src/Plugins/OrientationAnalysis/docs/AlignSectionsMisorientationFilter.md b/src/Plugins/OrientationAnalysis/docs/AlignSectionsMisorientationFilter.md index 0e9674d464..26f30ef822 100644 --- a/src/Plugins/OrientationAnalysis/docs/AlignSectionsMisorientationFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/AlignSectionsMisorientationFilter.md @@ -45,6 +45,20 @@ In this new structure, what follows is what the created structures represent: In previous versions a file would have been produced instead. If you wish to recreate this, you can write the Attribute Matrix as a CSV/Text file. +## Algorithm + +### In-Core Path + +For each pair of adjacent Z-sections, the algorithm computes the misorientation between voxels across the section boundary. It tests candidate X-Y shifts to find the shift that minimizes the total misorientation between the two sections. All voxel comparisons use direct array indexing with `operator[]`. + +### Out-of-Core Path + +Reads pairs of adjacent Z-slices into local memory buffers using `copyIntoBuffer()`. All misorientation comparisons for a given slice pair operate entirely on the in-memory buffers. This converts what would otherwise be random cross-slice element access into sequential bulk reads, avoiding chunk thrashing when data is stored on disk in compressed chunks. + +### Performance + +The OOC optimization matters most for large datasets that exceed available RAM. By reading entire Z-slices in bulk rather than accessing individual voxels across slice boundaries, the algorithm avoids repeatedly decompressing the same disk chunks. For in-memory datasets, the two paths produce identical results with negligible overhead difference. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/AlignSectionsMutualInformationFilter.md b/src/Plugins/OrientationAnalysis/docs/AlignSectionsMutualInformationFilter.md index 9fe3485150..a6c77122e4 100644 --- a/src/Plugins/OrientationAnalysis/docs/AlignSectionsMutualInformationFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/AlignSectionsMutualInformationFilter.md @@ -49,6 +49,20 @@ In this new structure, what follows is what the created structures represent: In previous versions a file would have been produced instead. If you wish to recreate this, you can write the Attribute Matrix as a CSV/Text file. +## Algorithm + +### In-Core Path + +Aligns Z-sections by maximizing the mutual information of orientation data between adjacent slices. The algorithm segments each slice into temporary features, bins the orientations, and computes joint and marginal histograms to evaluate mutual information at each candidate shift position. All orientation and feature ID data is accessed through direct array indexing with `operator[]`. + +### Out-of-Core Path + +Reads the orientation and phase data for each pair of adjacent Z-slices into local memory buffers using `copyIntoBuffer()` before computing histograms. This eliminates per-voxel out-of-core reads during the histogram binning and mutual information calculation, replacing them with two sequential bulk reads per slice pair. + +### Performance + +The OOC optimization matters most for large datasets that exceed available RAM. Histogram computation requires visiting every voxel in both slices multiple times (once per candidate shift), so eliminating per-element OOC access prevents repeated decompression of the same disk chunks. For in-memory datasets, the two paths produce identical results with negligible overhead difference. + % Auto generated parameter table will be inserted here ## References diff --git a/src/Plugins/OrientationAnalysis/docs/BadDataNeighborOrientationCheckFilter.md b/src/Plugins/OrientationAnalysis/docs/BadDataNeighborOrientationCheckFilter.md index a79b203db1..05b9605c40 100644 --- a/src/Plugins/OrientationAnalysis/docs/BadDataNeighborOrientationCheckFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/BadDataNeighborOrientationCheckFilter.md @@ -41,6 +41,32 @@ since there are no neighbors is the +-Z directions. Only the *Mask* value defining the cell as *good* or *bad* is changed. No other cell level array is modified. +## Algorithm + +The algorithm operates in a multi-level iterative scheme, starting at level 6 (all 6 face-neighbors must agree) and decrementing to the user-specified *Required Number of Neighbors*. At each level, bad voxels are flipped to good if they have at least that many good face-neighbors with matching crystallographic orientation (misorientation below the tolerance). Starting strict and relaxing ensures high-confidence flips happen first, which can cascade to enable additional flips. + +### In-Core Path (BadDataNeighborOrientationCheckWorklist) + +When all arrays reside in contiguous in-memory storage, the algorithm uses a two-phase worklist approach: + +1. **Phase 1 (Initial count)**: A single linear scan counts matching good face-neighbors for every bad voxel, storing the count in a per-voxel array. +2. **Phase 2 (Worklist propagation)**: For each level, a deque is seeded with all bad voxels meeting the threshold. As each voxel is flipped, its still-bad neighbors' counts are incremented. If a neighbor's count now meets the threshold, it is enqueued. This breadth-first flood-fill processes each voxel at most once per level, achieving O(flipped) amortized cost. + +### Out-of-Core Path (BadDataNeighborOrientationCheckScanline) + +When any of the quaternion, mask, or phase arrays are backed by chunked (OOC) disk storage, the algorithm uses a 3-slice rolling window over the Z axis: + +1. Three Z-slices of quaternions, phases, and mask data are maintained in memory (previous, current, next). +2. For each bad voxel in the current slice, the count of matching good face-neighbors is recomputed on-the-fly using the rolling window buffers. +3. If a voxel is flipped, the mask change is written back to the OOC store per-slice via `copyFromBuffer()`. +4. The window shifts forward one Z-slice at a time, with only one new slice loaded per step. + +This approach trades recomputation (no persistent neighbor-count array) for strictly sequential I/O that avoids the random-access chunk thrashing that would occur with the worklist variant's BFS pattern. + +### Performance + +The in-core worklist variant is significantly faster for datasets that fit in RAM because each voxel is processed at most once per level (O(flipped) cost vs. O(N * passes) for the scanline variant). The OOC scanline variant is slower in absolute terms but avoids catastrophic performance degradation on disk-backed datasets where the worklist's random access pattern would trigger continuous chunk load/evict cycles. Memory usage is O(N) for the worklist variant vs. O(3 * sliceSize) for the scanline variant. + ## Example Data | Example Input Image | Example Output Image | diff --git a/src/Plugins/OrientationAnalysis/docs/CAxisSegmentFeaturesFilter.md b/src/Plugins/OrientationAnalysis/docs/CAxisSegmentFeaturesFilter.md index 6983636597..95049db8f6 100644 --- a/src/Plugins/OrientationAnalysis/docs/CAxisSegmentFeaturesFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/CAxisSegmentFeaturesFilter.md @@ -47,6 +47,22 @@ is to still use the 6 face neighbors ("Face Only") in order to stay consistent w |:--:|:--:| | ![Shared Edges & Points With Disconnected Region - "Face Only"](Images/SegmentFeatures/combination_face_only.png) | ![Shared Edges & Points With Disconnected Region - "All Connected"](Images/SegmentFeatures/combination_all_connected.png) | +## Algorithm + +This filter segments voxels into features based on c-axis alignment, targeting hexagonal crystal systems. Voxels whose c-axes (the <001> crystallographic direction) are aligned within a user-defined tolerance are grouped into the same feature. + +### In-Core Path + +Uses a BFS-style flood fill where seed voxels are compared to their neighbors by computing the c-axis misalignment via direct array access with `operator[]`. Neighbors within tolerance are added to the current feature and queued for further expansion. + +### Out-of-Core Path + +Adapted to use sequential data access through the `SegmentFeatures` base class OOC support. The base class manages bulk I/O so that the flood-fill algorithm can proceed without triggering per-voxel OOC reads across chunk boundaries. + +### Performance + +The OOC optimization matters most for large datasets that exceed available RAM. Flood-fill naturally exhibits random access patterns as it grows regions across Z-slices, which can cause severe chunk thrashing with compressed on-disk storage. The OOC path mitigates this by leveraging the base class sequential access strategy. For in-memory datasets, the two paths produce identical results with negligible overhead difference. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeAvgCAxesFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeAvgCAxesFilter.md index ee93e24852..5bb5f4ec6a 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeAvgCAxesFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeAvgCAxesFilter.md @@ -24,6 +24,22 @@ This filter will error out if **ALL** phases are non-hexagonal. Any non-hexagona The output is a direction vector for each feature. +## Algorithm + +For each element, the quaternion orientation is converted to an active rotation matrix (by transposing the passive orientation matrix), which is then applied to the crystallographic c-axis direction <001> to find the c-axis location in the sample reference frame. These rotated c-axis vectors are accumulated per feature using a running average that checks sign consistency (flipping vectors whose dot product with the running average is negative). After all elements are processed, each feature's accumulated c-axis is normalized to produce the final average direction. + +### In-Core Path + +All cell-level arrays (feature IDs, phases, quaternions) and the feature-level output array are accessed through the AbstractDataStore API. Crystal structures are validated to ensure at least one hexagonal phase is present. + +### Out-of-Core Path + +Cell-level data is read in sequential 4096-tuple chunks via `copyIntoBuffer`, with separate buffers for feature IDs, cell phases, and quaternions. Feature-level accumulation is performed in a local buffer (`avgCAxesCache`) that is bulk-read at the start and bulk-written back via `copyFromBuffer` after the normalization pass. The ensemble-level crystal structures array is cached locally at startup to avoid per-element OOC overhead. + +### Performance + +The chunked sequential reads convert what would be millions of individual virtual dispatch calls into a small number of bulk I/O operations. Since the feature-level buffer is proportional to the number of grains (thousands) rather than voxels (millions), it fits comfortably in memory even when cell data is on disk. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeAvgOrientationsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeAvgOrientationsFilter.md index b3e261622d..dcc9d9a0b8 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeAvgOrientationsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeAvgOrientationsFilter.md @@ -68,6 +68,26 @@ These values may be exposed as user-configurable parameters in a future release. - **Features with zero elements:** Features with no elements (phase <= 0 for all voxels) will have their output arrays initialized to NaN (for vMF/Watson) or identity quaternion / zero Euler angles (for Rodrigues). - **Phase indexing:** The filter requires that phase values be > 0 for elements to be included in the averaging. Phase index 0 is reserved for "Unknown" in the Crystal Structures array and is always skipped. +## Algorithm + +This filter supports three independent averaging methods that can be enabled in any combination. Each method accumulates per-element quaternion data grouped by feature ID, then produces feature-level outputs. + +### In-Core Path + +**Rodrigues Average:** Iterates over all elements once, accumulating quaternion sums into a feature-level buffer. For each element, the voxel quaternion is rotated to the nearest symmetry-equivalent orientation of the running average to handle the periodicity of orientation space. After accumulation, each feature's summed quaternion is normalized, forced into the positive hemisphere, and converted to Euler angles. + +**Von Mises-Fisher / Watson Average:** A preliminary pass counts elements per feature and maps feature IDs to phases. Then a `ParallelDataAlgorithm` processes features in parallel. For each feature, all element quaternions are collected, reduced to the fundamental zone, and passed to the EbsdLib `DirectionalStats` EM algorithm to estimate the mean orientation (mu) and concentration parameter (kappa). + +### Out-of-Core Path + +The Rodrigues average reads cell-level arrays (feature IDs, phases, quaternions) in sequential 64K-tuple chunks via `copyIntoBuffer`, accumulating into local `std::vector` buffers that hold only the feature-level data. Crystal structures are cached locally from the tiny ensemble-level array. Final results are bulk-written to the DataStore via `copyFromBuffer`, eliminating random-access overhead on potentially disk-backed stores. + +The vMF/Watson path similarly reads cell-level data through the AbstractDataStore API and caches crystal structures locally. + +### Performance + +The chunked Rodrigues path avoids per-element virtual dispatch on the DataStore, which is critical when cell-level data exceeds available RAM and falls back to HDF5-chunked storage. The feature-level buffers remain in-memory because feature counts are orders of magnitude smaller than cell counts. The vMF/Watson path benefits from parallel feature processing since each feature's EM computation is independent. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeCAxisLocationsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeCAxisLocationsFilter.md index 9183413de0..c9baa4ac24 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeCAxisLocationsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeCAxisLocationsFilter.md @@ -10,6 +10,22 @@ This **Filter** determines the direction of the C-axis for each **Elemen *Note:* This **Filter** will only work properly for *Hexagonal* materials. The **Filter** does not apply any symmetry operators because there is only one c-axis (<001>) in *Hexagonal* materials and thus all symmetry operators will leave the c-axis in the same position in the sample *reference frame*. However, in *Cubic* materials, for example, the {100} family of directions are all equivalent and the <001> direction will change location in the *sample reference frame* when symmetry operators are applied. +## Algorithm + +For each element, the quaternion is converted to an orientation matrix. The matrix is transposed (converting the passive rotation to an active rotation) and multiplied by the crystallographic c-axis <001> to produce the c-axis direction in the sample reference frame. The result is normalized, and if the z-component is negative the vector is flipped to ensure consistent hemisphere placement. Non-hexagonal phases produce NaN output values. + +### In-Core Path + +Input quaternions, cell phases, and the output c-axis locations array are accessed through the AbstractDataStore API. + +### Out-of-Core Path + +Cell-level data is processed in sequential 64K-tuple chunks. Quaternions (4 components) and phases (1 component) are bulk-read via `copyIntoBuffer`, the c-axis is computed for each element in the chunk, and results (3 components) are bulk-written via `copyFromBuffer`. The ensemble-level crystal structures array is cached locally at startup. + +### Performance + +The simple per-element transform (quaternion to rotation matrix to c-axis vector) is compute-light, so I/O dominates the cost for OOC data. Chunked sequential access converts millions of virtual dispatch calls into a small number of contiguous reads and writes. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeFeatureNeighborCAxisMisalignmentsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeFeatureNeighborCAxisMisalignmentsFilter.md index d4ea2cc0b1..5b97cafcb6 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeFeatureNeighborCAxisMisalignmentsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeFeatureNeighborCAxisMisalignmentsFilter.md @@ -20,6 +20,22 @@ There are 2 outputs from this filter: Results from this filter can differ from its original version in DREAM.3D 6.5.171 by around 0.0001. This version uses double precision and Eigen for matrix operations which account for the differences in output. +## Algorithm + +For each feature, the algorithm converts the feature's average quaternion to an active rotation matrix and applies it to the <001> c-axis to find the c-axis direction in the sample reference frame. It then iterates over the feature's neighbor list, performing the same c-axis computation for each neighbor. The misalignment angle between the two c-axes is computed via the arc-cosine of their dot product, folded to [0, 90] degrees. Only neighbor pairs where both features are hexagonal and share the same phase contribute valid values; mismatched pairs are assigned NaN. If requested, the average misalignment across all valid neighbors is also computed per feature. + +### In-Core Path + +Feature-level arrays (phases, average quaternions) and ensemble-level arrays (crystal structures) are accessed through the AbstractDataStore API. NeighborList data structures provide per-feature neighbor information. + +### Out-of-Core Path + +All feature-level arrays (phases, average quaternions) and the ensemble-level crystal structures are bulk-read into local `std::vector` caches at startup via `copyIntoBuffer`. The per-feature loop then operates entirely on these local caches with zero OOC virtual dispatch overhead. The optional average misalignment output is accumulated in a local buffer and bulk-written via `copyFromBuffer` at the end. + +### Performance + +Because this filter operates on feature-level data (thousands of entries, not millions of cells), the entire working set fits comfortably in memory. The local caching eliminates any OOC overhead from the hot nested loop over features and their neighbors. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceCAxisMisorientationsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceCAxisMisorientationsFilter.md index 197a9b68cf..341952da41 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceCAxisMisorientationsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceCAxisMisorientationsFilter.md @@ -12,6 +12,24 @@ This filter requires at least one Hexagonal crystal structure phase (Hexagonal-L Results from this filter can differ from its original version in DREAM3D 6.6 by around 0.0001. This version uses double precision in part of its calculation to improve agreement and accuracy between platforms (notably ARM). +## Algorithm + +For each cell, the quaternion is converted to an active rotation matrix (transpose of the passive orientation matrix) and applied to the <001> c-axis to find the c-axis direction in the sample reference frame. The angle between this cell-level c-axis and the feature's average c-axis is computed. Angles exceeding 90 degrees are folded to (180 - angle) to account for the antipodal symmetry of hexagonal c-axes. The per-cell misorientation values are then used to compute the average and population standard deviation per feature. + +### In-Core Path + +Cell-level arrays are iterated directly. Crystal structure validation ensures at least one hexagonal phase is present. + +### Out-of-Core Path + +Cell-level data (feature IDs, phases, quaternions) is processed one Z-slice at a time. For each slice, all input arrays are bulk-read via `copyIntoBuffer` into pre-allocated slice buffers, the misorientation is computed for every cell in the slice, and results are bulk-written via `copyFromBuffer`. The feature-level average c-axes array is cached locally at startup. Crystal structures are also cached from the ensemble-level array. + +A second Z-slice pass re-reads the cell-level feature IDs and the just-written misorientation output to compute the population standard deviation per feature. + +### Performance + +The Z-slice processing pattern is well-suited for ImageGeom data where HDF5 chunks are often aligned by slice. Each slice is a single contiguous range in the linear cell index, so the bulk reads are sequential and cache-friendly. The two-pass approach (mean then standard deviation) avoids storing all cell values in memory simultaneously. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceMisorientationsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceMisorientationsFilter.md index 1d9b11307f..0e99bde843 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceMisorientationsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeFeatureReferenceMisorientationsFilter.md @@ -45,6 +45,28 @@ feature boundary, and use that voxel's orientation as the **reference orientatio ![ComputeFeatureReferenceMisorientations_1.png](Images/ComputeFeatureReferenceMisorientations_1.png) +## Algorithm + +The algorithm calculates the crystallographic misorientation angle between each cell's quaternion and a reference orientation for its parent feature. The misorientation is computed using the LaueOps symmetry operators for the cell's crystal structure, and the result is stored in degrees. + +When using the **Average Feature Orientation** reference, the pre-computed average quaternions are looked up per feature. When using the **Orientation Farthest from Feature Boundary** reference, a preliminary pass finds the cell with the maximum Euclidean distance from the grain boundary for each feature, and that cell's quaternion is used as the reference. + +After computing per-cell misorientations, the filter also calculates the average misorientation across all cells belonging to each feature. + +### In-Core Path + +Input cell-level arrays (feature IDs, phases, quaternions) and the reference data (average quaternions or grain boundary distances) are accessed through the AbstractDataStore API. The per-cell misorientation output is written directly. + +### Out-of-Core Path + +All cell-level arrays are read in sequential 64K-tuple chunks via `copyIntoBuffer`. Crystal structures and average quaternions are cached locally at startup since they are ensemble-level and feature-level arrays respectively. For the boundary-distance reference mode, the center voxel identification pass also uses chunked reads of feature IDs and distance arrays. + +Per-cell misorientation results are accumulated in a local buffer and bulk-written via `copyFromBuffer` one chunk at a time. Feature-level sums and counts are maintained in local vectors. + +### Performance + +The two-pass chunked design (center-finding pass for mode 1, then the main misorientation pass) ensures that cell-level data is always read sequentially. This avoids random page faults on HDF5-chunked DataStores and reduces I/O from millions of individual accesses to a few hundred bulk reads. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeGBCDFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeGBCDFilter.md index 5bb54a2536..58e533fbe0 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeGBCDFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeGBCDFilter.md @@ -8,6 +8,24 @@ Statistics (Crystallography) This **Filter** computes the 5D grain boundary character distribution (GBCD) for a **Triangle Geometry**, which is the relative area of grain boundary for a given misorientation and normal. The GBCD can be visualized by using either the **Write GBCD Pole Figure (GMT)** or the **Write GBCD Pole Figure (VTK)** **Filters**. +## Algorithm + +The filter computes the 5D grain boundary character distribution by iterating over all triangle faces in chunks. For each triangle, the Euler angles of the two adjacent features are used with all pairs of crystal symmetry operators to compute the symmetric misorientation and the crystal normal direction. These are mapped to a 5D GBCD bin index. The triangle's area is accumulated into the corresponding bin. After all triangles are processed, the histogram is normalized by total face area per phase to produce a distribution in multiples of the random distribution (MRD). + +### In-Core Path + +Feature-level arrays (Euler angles, phases) and face-level arrays (labels, normals, areas) are accessed through the AbstractDataStore API. Triangle processing is parallelized using `ParallelDataAlgorithm` within each chunk. + +### Out-of-Core Path + +Feature-level Euler angles, phases, and ensemble-level crystal structures are bulk-read into local caches at startup via `copyIntoBuffer`. Triangle-level arrays (face labels, normals, areas) are read in chunks of 50,000 triangles via `copyIntoBuffer`. The parallel GBCD bin computation for each chunk operates on raw pointer offsets into these local buffers, with zero OOC virtual dispatch. + +The full GBCD output array is accumulated in a local `std::vector` buffer (sized by bin resolution, not cell count) and bulk-written to the DataStore via `copyFromBuffer` after normalization. + +### Performance + +The dominant cost is the O(triangles * symmetry_ops^2) bin computation. By caching feature data locally and chunk-reading triangle data, the algorithm avoids OOC overhead in the triple-nested symmetry loop. The GBCD output buffer size depends on the bin resolution parameter, not the number of cells, so it remains manageable in memory. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeGBCDMetricBasedFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeGBCDMetricBasedFilter.md index 092dc86566..010b379cf5 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeGBCDMetricBasedFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeGBCDMetricBasedFilter.md @@ -55,6 +55,24 @@ The *Limiting Distances* parameter selects the maximum angular deviations used w - **7 deg. Misorientations; 7 deg. Plane Inclinations [5]**: Use a 7-degree radius for both misorientations and plane inclinations. - **8 deg. Misorientations; 8 deg. Plane Inclinations [6]**: Use an 8-degree radius for both misorientations and plane inclinations. +## Algorithm + +The filter computes a section of the GBCD for a fixed misorientation using a metric-based approach. First, boundary segments whose misorientations are within a limiting distance of the fixed misorientation are selected. Then the distribution is probed at sampling points generated by a golden section spiral on the hemisphere. For each sampling direction, the areas of selected triangles whose normals fall within the plane-inclination radius are summed. The result is normalized to multiples of the random distribution (MRD), and statistical errors are computed from the distribution values, number of grain boundaries, and the volume restricted by the resolution parameters. + +### In-Core Path + +Feature-level arrays (Euler angles, phases, feature-face labels), face-level arrays (labels, normals, areas), and node types are accessed through the AbstractDataStore API. Triangle selection and distribution probing are both parallelized using `ParallelDataAlgorithm`. + +### Out-of-Core Path + +Feature-level Euler angles, phases, ensemble-level crystal structures, and feature-face labels are bulk-read into local caches at startup via `copyIntoBuffer`. Triangle-level face labels and areas are chunk-read (50,000 triangles at a time) for the total face area accumulation pass. + +The parallel `TrianglesSelector` worker operates on the locally cached Euler and phase data (via raw pointers) while reading face-level arrays through the DataStore API. The parallel `ProbeDistribution` worker operates entirely on the in-memory `selectedTriangles` collection and sampling point vectors. + +### Performance + +The local caching of feature-level data is essential because the triangle selection loop performs random feature lookups (by face label) within the symmetry operator nested loop. The selected triangles are stored in a TBB concurrent vector (or std::vector in single-threaded mode) and later iterated sequentially during the probing phase. The two-phase design (selection then probing) limits peak memory to the selected subset rather than all triangles. + % Auto generated parameter table will be inserted here ## References diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeGBCDPoleFigureFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeGBCDPoleFigureFilter.md index 9b897bf055..55c9114ec1 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeGBCDPoleFigureFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeGBCDPoleFigureFilter.md @@ -8,10 +8,36 @@ IO (Output) This **Filter** creates a pole figure from the Grain Boundary Character Distribution (GBCD) data. The user must select the relevant phase for which to generate the pole figure by entering the *phase index*. +The GBCD is a 5-dimensional histogram that captures the statistical distribution of grain boundary planes as a function of the misorientation between the two grains meeting at each boundary. This filter extracts a 2D stereographic projection (pole figure) from that 5D distribution for a specific misorientation and crystal phase. + ![Regular Grid Visualization of the Small IN100 GBCD results](Images/Small_IN00_GBCD_RegularGrid.png) ![Using ParaView's Threshold filter + Cells to Points + Delaunay2D Filters](Images/Small_IN100_GBCD_Delaunay2D.png) +## Algorithm + +For each pixel (x, y) in the output square image, the filter: + +1. Performs inverse stereographic projection to obtain a unit-sphere direction representing the boundary-plane normal. +2. Iterates over all pairs of crystal symmetry operators (nSym x nSym) for the selected Laue class. +3. For each symmetry pair, computes the symmetrically-equivalent misorientation in both crystal reference frames. +4. If the equivalent misorientation falls within the fundamental zone (all three Euler angles < pi/2), the corresponding 5D GBCD bin is looked up and the intensity is accumulated. +5. The pixel intensity is the average GBCD value across all valid symmetry-pair lookups. + +Pixels outside the unit circle of the stereographic projection are left at zero intensity. + +### In-Core Path (ComputeGBCDPoleFigureDirect) + +When the GBCD array resides in contiguous in-memory storage, the algorithm caches the entire GBCD array (all phases) into a local heap buffer, then uses `ParallelData2DAlgorithm` to compute pixel intensities in parallel across the output image grid. The parallel workers access only the cached raw pointer, so no `DataStore` access occurs in the hot loop. + +### Out-of-Core Path (ComputeGBCDPoleFigureScanline) + +When the GBCD array is backed by chunked (OOC) disk storage, the algorithm caches only the single-phase slice of the GBCD needed for the selected phase of interest. This is the critical optimization: for a typical GBCD, one phase slice is on the order of 100K-500K float64 elements, compared to millions for the full multi-phase array. Once the phase slice is cached, pixel computation proceeds in parallel identically to the in-core path. + +### Performance + +Both paths use multi-threaded parallel pixel computation. The difference is only in how much GBCD data is loaded into memory: the in-core path loads everything; the OOC path loads only the phase of interest. For single-phase analyses, the performance is nearly identical. For multi-phase datasets stored out-of-core, the OOC path avoids loading irrelevant phase data and reduces memory consumption proportionally to the number of phases. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeIPFColorsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeIPFColorsFilter.md index 5a9861f4e4..435897240e 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeIPFColorsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeIPFColorsFilter.md @@ -26,6 +26,30 @@ IPF Legends for most all of the Laue classes can be found in the Data/Orientatio ![Example Data Set](Images/IPFColor_1.png) +## Algorithm + +For each voxel, the filter converts the stored Euler angles (phi1, Phi, phi2) into an orientation matrix, transforms the user-specified reference direction from the sample frame into the crystal frame, and maps the resulting crystal-frame direction to a position on the inverse pole figure triangle for the voxel's Laue symmetry class. That position determines the RGB color. + +Voxels that are masked out (bad data), that have an unindexed phase (phase 0), or whose phase ID exceeds the crystal structures ensemble array are colored black (0, 0, 0). If any voxels have out-of-range phase IDs, the filter returns an error with a count of affected voxels. + +### In-Core Path (ComputeIPFColorsDirect) + +When all arrays reside in contiguous in-memory storage, the algorithm uses `ParallelDataAlgorithm` to split the voxel range across threads. Each thread independently computes IPF colors for its assigned sub-range using direct `AbstractDataStore` access. + +### Out-of-Core Path (ComputeIPFColorsScanline) + +When any of the Euler-angle, phase, or IPF-color arrays are backed by chunked (OOC) disk storage, the algorithm switches to a sequential chunk-based approach. It processes voxels in fixed-size chunks of 65,536 tuples: + +1. Bulk-read Euler angles, phase IDs, and (optionally) the mask for the chunk via `copyIntoBuffer()`. +2. Compute IPF colors for every tuple in the chunk using the same `LaueOps::generateIPFColor()` logic. +3. Bulk-write the computed RGB colors back via `copyFromBuffer()`. + +This linear access pattern reads each OOC disk chunk at most once, avoiding the random-access chunk thrashing that would occur if the in-core parallel path accessed OOC stores concurrently. + +### Performance + +The in-core path achieves near-linear multi-threaded speedup. The OOC path is single-threaded but disk-I/O-limited, with throughput determined by sequential read/write bandwidth rather than random-access latency. For datasets that fit in RAM, the in-core path is significantly faster; for datasets that exceed available memory, the OOC path avoids catastrophic slowdowns from virtual memory paging or chunk eviction. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeKernelAvgMisorientationsFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeKernelAvgMisorientationsFilter.md index c6e615cc2a..5884ea5076 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeKernelAvgMisorientationsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeKernelAvgMisorientationsFilter.md @@ -15,6 +15,22 @@ The calculation will **not** consider cells that belong to different 'feature Id *Note:* All **Cells** in the kernel are weighted equally during the averaging, though they are not equidistant from the central **Cell**. +## Algorithm + +For each cell in the ImageGeom, the algorithm examines all cells within a user-specified kernel radius in X, Y, and Z. Only neighbor cells that share the same feature ID as the center cell are included. The crystallographic misorientation angle between the center cell's quaternion and each qualifying neighbor's quaternion is computed using the appropriate LaueOps symmetry operators. The average of these misorientation angles is stored as the KAM value for the center cell. + +### In-Core Path + +All cell-level arrays (phases, feature IDs, quaternions) are accessed through the AbstractDataStore API. The output array is written directly. + +### Out-of-Core Path + +The algorithm processes data one Z-plane at a time. For each plane, a slab of input data spanning `[plane - kernelZ, plane + kernelZ]` is bulk-read via `copyIntoBuffer`. This slab contains all data needed for neighbor lookups of cells in the current plane. The crystal structures array is cached locally at startup. Output values for each plane are accumulated in a local buffer and bulk-written via `copyFromBuffer`. + +### Performance + +The slab-based approach is critical for KAM because each cell needs random access to its neighbors within the kernel radius. By reading the entire slab into memory, all neighbor lookups become local memory accesses rather than individual OOC page faults. The slab size is bounded by `(2 * kernelZ + 1) * sliceSize`, which is manageable even for large kernel radii. Sequential plane processing ensures the data is read in order through the volume. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ComputeTwinBoundariesFilter.md b/src/Plugins/OrientationAnalysis/docs/ComputeTwinBoundariesFilter.md index 5712cbb442..2df76ae22b 100644 --- a/src/Plugins/OrientationAnalysis/docs/ComputeTwinBoundariesFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ComputeTwinBoundariesFilter.md @@ -15,6 +15,26 @@ The *Output Type for Twin Boundaries Array* parameter controls the data type use - **boolean [0]**: Stores the twin boundary flag as a boolean array (true if the **Triangle** is a twin boundary, false otherwise). - **uint8 [1]**: Stores the twin boundary flag as an unsigned 8-bit integer array (1 if the **Triangle** is a twin boundary, 0 otherwise). +## Algorithm + +For each triangle face in the surface mesh, the algorithm checks whether the two adjacent features are in the same phase with a cubic crystal structure. If so, the misorientation between the two features' average quaternions is computed using all symmetry operator pairs. A face is flagged as a twin boundary if any symmetric equivalent produces a misorientation within the user-defined angle and axis tolerances of the Sigma 3 twin relationship (60 degrees about <111>). + +When coherence computation is enabled, the crystal direction parallel to the face normal is determined and compared with the misorientation axis. The minimum angular deviation across all valid symmetry pairs is stored as the incoherence value. + +### In-Core Path + +Feature-level arrays (phases, average quaternions) and face-level arrays (labels, normals) are accessed through the AbstractDataStore API. The twin boundary check is parallelized using `ParallelDataAlgorithm`. + +### Out-of-Core Path + +All input arrays are bulk-read into local `std::vector` caches at startup: ensemble-level crystal structures, feature-level phases and average quaternions, and face-level labels and normals. The parallel workers (`CalculateTwinBoundaryImpl` and `CalculateTwinBoundaryWithIncoherenceImpl`) operate entirely on these local vectors with zero OOC virtual dispatch in the hot loop. + +Output is accumulated into local vectors and bulk-written to the DataStores via `copyFromBuffer` after the parallel computation completes. + +### Performance + +Pre-caching all arrays into contiguous local vectors enables safe parallel execution without any thread contending for OOC page locks. Face-level data scales with surface area (not volume), so it typically fits in memory even for large datasets. The parallel twin boundary check across all symmetry operator pairs is the dominant compute cost and benefits from the contention-free data access. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/ConvertOrientationsFilter.md b/src/Plugins/OrientationAnalysis/docs/ConvertOrientationsFilter.md index 0b20a08880..53e1c95e76 100644 --- a/src/Plugins/OrientationAnalysis/docs/ConvertOrientationsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/ConvertOrientationsFilter.md @@ -90,6 +90,22 @@ If the angles fall outside of this range the **original** Euler Input data **WIL While every effort has been made to ensure the correctness of each transformation algorithm, certain situations may arise where the initial precision of the input data is not large enough for the algorithm to calculate an answer that is intuitive. The user should be acutely aware of their input data and if their data may cause these situations to occur. Combinations of Euler angles close to 0, 180 and 360 can cause these issues to be hit. For instance an Euler angle of [180, 56, 360] is symmetrically the same as [180, 56, 0] and due to calculation errors and round off errors converting that Euler angle between representations may not give the numerical answer the user was anticipating but will give a symmetrically equivalent angle. +## Algorithm + +The filter converts each element's orientation from the input representation to the output representation using the EbsdLib orientation conversion library. The conversion is performed element-by-element through a chain of intermediate representations as needed (e.g., Euler to Quaternion may go through an orientation matrix). Euler angle inputs are range-checked and clamped before conversion. The computation is parallelized using `ParallelDataAlgorithm` with a macro-generated converter class for each output type. + +### In-Core Path + +Input and output DataArrays are accessed through the AbstractDataStore API. The parallel converter reads input and writes output directly. + +### Out-of-Core Path + +Each parallel converter processes its assigned tuple range in 4096-tuple chunks. Within each chunk, input data is bulk-read via `copyIntoBuffer`, the orientation conversion is performed element-by-element on the local buffer, and results are bulk-written via `copyFromBuffer`. This chunked approach is embedded in the `OC_TBB_IMPL` macro that generates all eight converter classes. + +### Performance + +The chunked I/O within each parallel range avoids per-element virtual dispatch on the DataStore while preserving parallelism across ranges. Since the conversion is purely per-element with no neighbor dependencies, the chunks can be processed independently with excellent scaling. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/EBSDSegmentFeaturesFilter.md b/src/Plugins/OrientationAnalysis/docs/EBSDSegmentFeaturesFilter.md index 73e65dbedb..490944062d 100644 --- a/src/Plugins/OrientationAnalysis/docs/EBSDSegmentFeaturesFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/EBSDSegmentFeaturesFilter.md @@ -49,6 +49,22 @@ is to still use the 6 face neighbors ("Face Only") in order to stay consistent w |:--:|:--:| | ![Shared Edges & Points With Disconnected Region - "Face Only"](Images/SegmentFeatures/combination_face_only.png) | ![Shared Edges & Points With Disconnected Region - "All Connected"](Images/SegmentFeatures/combination_all_connected.png) | +## Algorithm + +This filter segments EBSD orientation data into crystallographic grains using flood-fill region growing. Voxels are grouped into the same feature if their misorientation is below a user-defined tolerance threshold. + +### In-Core Path + +Uses a BFS-style flood fill where seed voxels are compared to their neighbors via random array access with `operator[]`. Each neighbor whose misorientation falls within tolerance is added to the current feature and queued for further comparison. + +### Out-of-Core Path + +Adapted to use sequential data access through the `SegmentFeatures` base class OOC support. The base class manages bulk I/O so that the flood-fill algorithm can proceed without triggering per-voxel OOC reads across chunk boundaries. + +### Performance + +The OOC optimization matters most for large datasets that exceed available RAM. Flood-fill naturally exhibits random access patterns as it grows regions across Z-slices, which can cause severe chunk thrashing with compressed on-disk storage. The OOC path mitigates this by leveraging the base class sequential access strategy. For in-memory datasets, the two paths produce identical results with negligible overhead difference. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/MergeTwinsFilter.md b/src/Plugins/OrientationAnalysis/docs/MergeTwinsFilter.md index 2ce1719337..cbb0188a56 100644 --- a/src/Plugins/OrientationAnalysis/docs/MergeTwinsFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/MergeTwinsFilter.md @@ -10,6 +10,22 @@ Reconstruction (Grouping) This **Filter** groups neighboring **Features** that are in a twin relationship with each other (currently only FCC σ = 3 twins). The algorithm for grouping the **Features** is analogous to the algorithm for segmenting the **Features** - only the average orientation of the **Features** are used instead of the orientations of the individual **Elements**. The user can specify a tolerance on both the *axis* and the *angle* that defines the twin relationship (i.e., a tolerance of 1 degree for both tolerances would allow the neighboring **Features** to be grouped if their misorientation was between 59-61 degrees about an axis within 1 degree of <111>, since the Sigma 3 twin relationship is 60 degrees about <111>). +## Algorithm + +The algorithm uses a seed-and-grow approach analogous to feature segmentation. A random unassigned feature is selected as a seed and assigned a new parent ID. The seed's contiguous neighbors are examined: if the misorientation between the seed and a neighbor is within tolerance of the Sigma 3 twin relationship (60 degrees about <111>), the neighbor is added to the same parent group. This process repeats for newly grouped features until no more twins are found, then a new seed is selected. After grouping, cell-level parent IDs are assigned by looking up each cell's feature ID in the feature-to-parent map. + +### In-Core Path + +Feature-level arrays (phases, parent IDs, average quaternions, crystal structures) are accessed through the AbstractDataStore API. The contiguous neighbor list provides adjacency information. Cell-level arrays are written directly. + +### Out-of-Core Path + +The cell-level parent ID array is initialized in 64K-element chunks via `copyFromBuffer` to avoid a single large fill operation on an OOC store. After the feature grouping phase completes, the feature-to-parent map is cached locally. Cell-level feature IDs are then read in 64K-tuple chunks via `copyIntoBuffer`, translated to parent IDs using the local cache, and the results are bulk-written via `copyFromBuffer`. + +### Performance + +The feature-level grouping algorithm involves random access to feature arrays (phases, quaternions, parent IDs), but feature counts are small enough (thousands) that this does not cause OOC bottlenecks. The cell-level pass, which touches millions of voxels, uses sequential chunked I/O to avoid per-element OOC overhead. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/NeighborOrientationCorrelationFilter.md b/src/Plugins/OrientationAnalysis/docs/NeighborOrientationCorrelationFilter.md index 2c63cb2dbb..44a3277d8a 100644 --- a/src/Plugins/OrientationAnalysis/docs/NeighborOrientationCorrelationFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/NeighborOrientationCorrelationFilter.md @@ -39,6 +39,20 @@ These before and after images show how this filter can be used to "fill in" data This filter will copy all attribute data from neighboring cells into the target cell if the criteria is met. +## Algorithm + +### In-Core Path + +For each voxel, the algorithm computes the misorientation with its 6 face-sharing neighbors. If too many neighbors disagree (exceed the misorientation tolerance), the voxel is flagged for cleanup and its attributes are replaced with those of a suitable neighbor. All data access uses direct array indexing with `operator[]`. + +### Out-of-Core Path + +Uses a 3-slice Z rolling window (previous, current, next) to provide all neighbor access from memory. For each Z-slice processed, the algorithm loads the three relevant slices into local buffers using `copyIntoBuffer()`. After processing a slice, modified data is written back using `copyFromBuffer()`, and the window advances by shifting buffers and reading the next slice. This ensures that all 6-neighbor lookups are serviced from in-memory buffers rather than individual OOC element reads. + +### Performance + +The OOC optimization matters most for large datasets that exceed available RAM. Because each voxel requires access to neighbors in adjacent Z-slices, naive OOC access would decompress the same chunks repeatedly. The rolling window approach reads each slice at most three times total (as previous, current, and next), converting random access into predictable sequential I/O. For in-memory datasets, the two paths produce identical results with negligible overhead difference. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/docs/RotateEulerRefFrameFilter.md b/src/Plugins/OrientationAnalysis/docs/RotateEulerRefFrameFilter.md index 31a82711c7..53937575c2 100644 --- a/src/Plugins/OrientationAnalysis/docs/RotateEulerRefFrameFilter.md +++ b/src/Plugins/OrientationAnalysis/docs/RotateEulerRefFrameFilter.md @@ -8,6 +8,22 @@ Processing (Conversion) This **Filter** performs a passive rotation (Right hand rule) of the Euler Angles about a user defined axis. The *reference frame* is being rotated and thus the *Euler Angles* necessary to represent the same orientation must change to account for the new *reference frame*. The user can set an *angle* and an *axis* to define the rotation of the *reference frame*. +## Algorithm + +The filter constructs a rotation matrix from the user-specified axis and angle. For each element, the Euler angles are converted to an orientation matrix, multiplied by the rotation matrix (applying the passive reference frame rotation), re-normalized column-wise, and converted back to Euler angles. This is an in-place operation that modifies the input Euler angle array. + +### In-Core Path + +The Euler angle array is accessed through the AbstractDataStore API for both reading and writing. + +### Out-of-Core Path + +The Euler angle array is processed in sequential 64K-tuple chunks. Each chunk is bulk-read via `copyIntoBuffer`, the rotation is applied to all elements in the chunk, and the modified values are bulk-written back to the same location via `copyFromBuffer`. The rotation matrix is computed once at the start from the user-specified axis-angle. + +### Performance + +The per-element rotation (Euler to matrix, matrix multiply, matrix to Euler) is moderately compute-intensive, but I/O dominates for OOC data. The chunked read-modify-write pattern ensures sequential access through the array with minimal OOC overhead. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp index 2050d915a0..675df0df69 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.cpp @@ -25,6 +25,10 @@ AlignSectionsMisorientation::AlignSectionsMisorientation(DataStructure& dataStru // ----------------------------------------------------------------------------- AlignSectionsMisorientation::~AlignSectionsMisorientation() noexcept = default; +// ----------------------------------------------------------------------------- +// Entry point: delegates to the base AlignSections::execute() which handles +// the overall alignment pipeline (initialize shift arrays, call findShifts(), +// apply shifts to the geometry, and optionally subtract a linear background). // ----------------------------------------------------------------------------- Result<> AlignSectionsMisorientation::operator()() { @@ -37,9 +41,19 @@ Result<> AlignSectionsMisorientation::operator()() return execute(gridGeom.getDimensions(), m_InputValues->ImageGeometryPath); } +// ----------------------------------------------------------------------------- +// Computes optimal X-Y shifts for each pair of adjacent Z-slices by minimizing +// the fraction of sampled voxel pairs whose misorientation exceeds the user +// tolerance. +// +// Dispatch logic: if any of the input arrays (quats, cellPhases) are backed by +// an OOC DataStore, or if ForceOocAlgorithm() is set, the OOC-optimized +// findShiftsOoc() path is used instead. The check is done in a limited scope +// so the temporary references are destroyed before the in-core path begins. // ----------------------------------------------------------------------------- Result<> AlignSectionsMisorientation::findShifts(std::vector& xShifts, std::vector& yShifts) { + // OOC dispatch: check if any input array uses out-of-core storage { const auto& quatsCheck = m_DataStructure.getDataRefAs(m_InputValues->quatsArrayPath); const auto& cellPhasesCheck = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp index 6da4a2c036..eb797a04b0 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMisorientation.hpp @@ -14,32 +14,70 @@ namespace nx::core { /** - * @brief + * @struct AlignSectionsMisorientationInputValues + * @brief Holds all user-supplied parameters for the AlignSectionsMisorientation algorithm. */ struct ORIENTATIONANALYSIS_EXPORT AlignSectionsMisorientationInputValues { - DataPath ImageGeometryPath; - bool UseMask; - DataPath MaskArrayPath; + DataPath ImageGeometryPath; ///< Path to the IGridGeometry defining the 3D voxel grid + bool UseMask = false; ///< Whether to exclude masked voxels from the alignment cost function + DataPath MaskArrayPath; ///< Path to the Bool/UInt8 mask array (only used when UseMask is true) - float32 misorientationTolerance; - DataPath quatsArrayPath; - DataPath cellPhasesArrayPath; - DataPath crystalStructuresArrayPath; + float32 misorientationTolerance = 0.0f; ///< Maximum misorientation angle (degrees) below which a cell pair is "aligned" + DataPath quatsArrayPath; ///< Path to the Float32 quaternion array (4 components per cell) + DataPath cellPhasesArrayPath; ///< Path to the Int32 cell phases array + DataPath crystalStructuresArrayPath; ///< Path to the UInt32 crystal structures ensemble array - bool StoreAlignmentShifts; - DataPath AlignmentAMPath; - DataPath SlicesArrayPath; - DataPath RelativeShiftsArrayPath; - DataPath CumulativeShiftsArrayPath; + bool StoreAlignmentShifts = false; ///< Whether to write computed shifts into output DataArrays + DataPath AlignmentAMPath; ///< Path to the Attribute Matrix storing alignment shift output arrays + DataPath SlicesArrayPath; ///< Path to the UInt32 array recording which slices were compared + DataPath RelativeShiftsArrayPath; ///< Path to the Int64 array recording per-iteration relative X/Y shifts + DataPath CumulativeShiftsArrayPath; ///< Path to the Int64 array recording cumulative X/Y shifts }; /** - * @brief + * @class AlignSectionsMisorientation + * @brief Aligns consecutive Z-sections of an EBSD dataset by finding the X-Y + * shift that minimizes cross-section crystallographic misorientation. + * + * For each pair of adjacent Z-slices the algorithm evaluates a 7x7 grid of + * candidate X-Y shifts. At each candidate position, voxel pairs between the + * two slices are sampled (every 4th voxel in X and Y) and the fraction of + * pairs whose misorientation exceeds the user-specified tolerance is computed. + * The candidate with the lowest fraction becomes the new grid center, and the + * search repeats until convergence (a downhill-simplex-like iterative refinement). + * + * Misorientation is computed via EbsdLib LaueOps symmetry operators using the + * quaternion representation stored in the input data. Only voxels that share + * the same crystallographic phase are compared; cross-phase pairs are counted + * as misoriented. + * + * ## Out-of-Core (OOC) Optimization + * + * When the input quaternion or phase arrays are backed by an out-of-core + * (chunked, on-disk) DataStore, the in-core path would trigger thousands of + * random chunk decompressions per slice pair because the 7x7 convergence loop + * re-reads the same voxels many times. The OOC path (findShiftsOoc) instead: + * 1. Pre-loads the topmost Z-slice into a "reference" buffer via bulk + * copyIntoBuffer() -- one contiguous read per array. + * 2. For each subsequent slice, bulk-reads the "current" slice into a second + * buffer, then runs the identical convergence logic on the local buffers. + * 3. After convergence, swaps the current buffer into the reference slot + * (O(1) std::swap), so the next iteration reuses the data without re-reading. + * + * This reduces per-slice I/O from O(thousands of random chunk reads) to exactly + * 2 sequential bulk reads (phases + quats) plus an optional mask read. */ class ORIENTATIONANALYSIS_EXPORT AlignSectionsMisorientation : public AlignSections { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Handler for sending progress messages to the UI. + * @param shouldCancel Atomic flag checked between iterations to support cancellation. + * @param inputValues User-supplied parameters controlling the alignment behavior. + */ AlignSectionsMisorientation(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, AlignSectionsMisorientationInputValues* inputValues); ~AlignSectionsMisorientation() noexcept override; @@ -48,30 +86,45 @@ class ORIENTATIONANALYSIS_EXPORT AlignSectionsMisorientation : public AlignSecti AlignSectionsMisorientation& operator=(const AlignSectionsMisorientation&) = delete; AlignSectionsMisorientation& operator=(AlignSectionsMisorientation&&) noexcept = delete; + /** + * @brief Executes the section alignment algorithm. + * @return Result<> indicating success or any errors encountered during execution. + */ Result<> operator()(); protected: /** - * @brief This method finds the slice to slice shifts and should be implemented by subclasses - * @param xShifts - * @param yShifts - * @return Whether the x and y shifts were successfully found + * @brief Computes the optimal X-Y shift for each pair of adjacent Z-slices. + * + * Dispatches to findShiftsOoc() when the input arrays are backed by an + * out-of-core DataStore; otherwise runs the in-core path that accesses the + * DataArrays directly via operator[]. + * + * @param xShifts Output vector of cumulative X shifts per slice (size = numSlices). + * @param yShifts Output vector of cumulative Y shifts per slice (size = numSlices). + * @return Success or error result. */ Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; private: /** * @brief OOC-optimized variant of findShifts that buffers two adjacent Z-slices - * into local vectors before the convergence loop, eliminating per-tuple chunk thrashing. + * (quaternions, phases, and mask) into local std::vectors before the convergence + * loop, eliminating per-tuple chunk thrashing on out-of-core DataStores. + * + * Uses a double-buffering strategy: after convergence for a slice pair, the + * "current" buffer is swapped into the "reference" slot via std::swap, avoiding + * a redundant re-read for the next iteration. + * * @param xShifts Output vector of cumulative X shifts per slice. * @param yShifts Output vector of cumulative Y shifts per slice. * @return Success or error result. */ Result<> findShiftsOoc(std::vector& xShifts, std::vector& yShifts); - DataStructure& m_DataStructure; - const AlignSectionsMisorientationInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const AlignSectionsMisorientationInputValues* m_InputValues = nullptr; ///< Non-owning pointer to the user-supplied parameters + const std::atomic_bool& m_ShouldCancel; ///< Atomic flag for cooperative cancellation + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for sending progress messages to the UI }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp index 1a82433041..c94870eeb2 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/AlignSectionsMutualInformation.hpp @@ -14,30 +14,76 @@ namespace nx::core { +/** + * @struct AlignSectionsMutualInformationInputValues + * @brief Holds all user-supplied parameters for the AlignSectionsMutualInformation algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT AlignSectionsMutualInformationInputValues { - DataPath ImageGeometryPath; - bool UseMask; - DataPath MaskArrayPath; + DataPath ImageGeometryPath; ///< Path to the ImageGeom defining the 3D voxel grid + bool UseMask = false; ///< Whether to exclude masked voxels from alignment + DataPath MaskArrayPath; ///< Path to the Bool/UInt8 mask array (only used when UseMask is true) - float32 MisorientationTolerance; - DataPath QuatsArrayPath; - DataPath CellPhasesArrayPath; - DataPath CrystalStructuresArrayPath; + float32 MisorientationTolerance = 0.0f; ///< Misorientation tolerance (degrees) used to segment features within each Z-slice + DataPath QuatsArrayPath; ///< Path to the Float32 quaternion array (4 components per cell) + DataPath CellPhasesArrayPath; ///< Path to the Int32 cell phases array + DataPath CrystalStructuresArrayPath; ///< Path to the UInt32 crystal structures ensemble array - bool StoreAlignmentShifts; - DataPath AlignmentAMPath; - DataPath SlicesArrayPath; - DataPath RelativeShiftsArrayPath; - DataPath CumulativeShiftsArrayPath; + bool StoreAlignmentShifts = false; ///< Whether to write computed shifts into output DataArrays + DataPath AlignmentAMPath; ///< Path to the Attribute Matrix storing alignment shift output arrays + DataPath SlicesArrayPath; ///< Path to the UInt32 array recording which slices were compared + DataPath RelativeShiftsArrayPath; ///< Path to the Int64 array recording per-iteration relative X/Y shifts + DataPath CumulativeShiftsArrayPath; ///< Path to the Int64 array recording cumulative X/Y shifts }; /** - * @class + * @class AlignSectionsMutualInformation + * @brief Aligns consecutive Z-sections by maximizing mutual information of + * orientation-based feature segmentations between adjacent slices. + * + * Unlike the misorientation-based alignment which compares individual voxel + * orientations, this algorithm first segments each 2D Z-slice into temporary + * "features" (groups of voxels whose orientations are within MisorientationTolerance). + * It then computes the mutual information between the feature-ID maps of + * adjacent slices. Mutual information measures the statistical dependence + * between two discrete distributions -- here, the joint probability of + * feature-ID pairs vs. their marginal probabilities. Higher mutual information + * indicates better alignment because features in neighboring slices overlap + * more consistently. + * + * The shift search uses the same 7x7 iterative-refinement approach as + * AlignSectionsMisorientation, but maximizes mutual information instead of + * minimizing misorientation count. + * + * ## Out-of-Core (OOC) Optimization + * + * The original algorithm stored all per-slice feature IDs in a single + * totalPoints-sized array and pre-segmented every slice up front, requiring + * the full quaternion, phase, and feature-ID arrays to be resident in memory. + * The optimized version uses a rolling 2-slice approach: + * 1. Bulk-reads one Z-slice of quaternions, phases, and mask via + * copyIntoBuffer() into local std::vectors. + * 2. Flood-fills the slice locally via formFeaturesForSlice() to produce + * a per-slice feature-ID vector (indices 0 to sliceVoxels-1). + * 3. Computes mutual information using only the current and reference + * feature-ID vectors. + * 4. After convergence, swaps the current feature-ID vector into the + * reference slot (O(1) std::swap) for the next iteration. + * + * This reduces memory from O(totalVoxels) to O(2 * sliceVoxels) and ensures + * each slice's data is read exactly once via a single sequential bulk I/O + * operation, eliminating chunk thrashing on OOC DataStores. */ class ORIENTATIONANALYSIS_EXPORT AlignSectionsMutualInformation : public AlignSections { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Handler for sending progress messages to the UI. + * @param shouldCancel Atomic flag checked between iterations to support cancellation. + * @param inputValues User-supplied parameters controlling the alignment behavior. + */ AlignSectionsMutualInformation(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, AlignSectionsMutualInformationInputValues* inputValues); ~AlignSectionsMutualInformation() noexcept override; @@ -47,36 +93,60 @@ class ORIENTATIONANALYSIS_EXPORT AlignSectionsMutualInformation : public AlignSe AlignSectionsMutualInformation& operator=(const AlignSectionsMutualInformation&) = delete; AlignSectionsMutualInformation& operator=(AlignSectionsMutualInformation&&) noexcept = delete; + /** + * @brief Executes the mutual-information-based section alignment algorithm. + * @return Result<> indicating success or any errors encountered during execution. + */ Result<> operator()(); protected: + /** + * @brief Computes the optimal X-Y shift for each pair of adjacent Z-slices + * by maximizing mutual information of per-slice feature segmentations. + * + * Uses a rolling 2-slice buffering strategy: each Z-slice is bulk-read once, + * flood-filled into a local feature-ID vector, and then the current vector + * is swapped into the reference slot after convergence. + * + * @param xShifts Output vector of cumulative X shifts per slice (size = numSlices). + * @param yShifts Output vector of cumulative Y shifts per slice (size = numSlices). + * @return Success or error result. + */ Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; private: /** * @brief Flood-fills a single Z-slice to identify features based on - * misorientation tolerance. Uses local indices (0 to sliceVoxels-1). - * Works for both in-core and OOC paths since the caller provides - * pre-buffered slice data. - * @param quats Pointer to the slice's quaternion data (4 components per voxel). - * @param phases Pointer to the slice's phase data. + * misorientation tolerance, operating on pre-buffered slice data. + * + * Uses a region-growing algorithm: starting from an unassigned seed voxel, + * expands to 4-connected in-plane neighbors (up, down, left, right) whose + * misorientation with the current voxel is below the tolerance. Each + * connected component receives a unique feature ID. + * + * This method works identically for both in-core and OOC paths because the + * caller provides pre-buffered raw pointers to the slice's data, so no + * DataStore access occurs inside the flood fill. + * + * @param quats Pointer to the slice's quaternion data (4 float32 components per voxel). + * @param phases Pointer to the slice's phase data (1 int32 per voxel). * @param mask Pointer to the slice's mask data (nullptr if mask is not used). * @param featureIds Output vector of per-voxel feature IDs (must be pre-zeroed, size = dimX*dimY). * @param dimX Number of voxels in X dimension. * @param dimY Number of voxels in Y dimension. * @param misorientationTolerance Misorientation tolerance in radians. * @param useMask Whether to use the mask array. - * @param orientationOps Laue orientation operators. - * @param crystalStructures Crystal structure IDs indexed by phase. - * @return The feature count for this slice. + * @param orientationOps Laue orientation operators for symmetry-aware misorientation. + * @param crystalStructures Crystal structure IDs indexed by phase number. + * @return The number of features found in this slice (feature IDs run from 1 to return-value-1). */ int32 formFeaturesForSlice(const float32* quats, const int32* phases, const uint8* mask, std::vector& featureIds, int64 dimX, int64 dimY, float32 misorientationTolerance, bool useMask, const std::vector& orientationOps, const std::vector& crystalStructures); - DataStructure& m_DataStructure; - const AlignSectionsMutualInformationInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const AlignSectionsMutualInformationInputValues* m_InputValues = nullptr; ///< Non-owning pointer to the user-supplied parameters + const std::atomic_bool& m_ShouldCancel; ///< Atomic flag for cooperative cancellation + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for sending progress messages to the UI }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp index 92bf066b3c..7d5b31a1c1 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.cpp @@ -28,8 +28,24 @@ const std::atomic_bool& BadDataNeighborOrientationCheck::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Dispatches the neighbor orientation check to the appropriate algorithm + * based on the storage type of the quaternion, mask, and phase arrays. + * + * The in-core path (Worklist) is preferred when all data is in RAM because its + * worklist-driven propagation has O(flipped) amortized cost. The OOC path + * (Scanline) is used when any array is chunked on disk, because the Worklist + * variant's random-access deque pattern would cause catastrophic chunk thrashing. + * + * Note: the in-core algorithm is BadDataNeighborOrientationCheckWorklist (not "Direct"), + * because the original linear-scan approach was replaced with a more efficient + * worklist-based propagation algorithm. + */ Result<> BadDataNeighborOrientationCheck::operator()() { + // Retrieve raw IDataArray pointers for storage-type inspection by DispatchAlgorithm. + // These are only used for the AnyOutOfCore() check -- the actual typed access + // happens inside the selected algorithm class. auto* quatsArray = m_DataStructure.getDataAs(m_InputValues->QuatsArrayPath); auto* maskArray = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); auto* phasesArray = m_DataStructure.getDataAs(m_InputValues->CellPhasesArrayPath); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp index 5a65acab3d..c8d7958083 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheck.hpp @@ -9,23 +9,55 @@ namespace nx::core { +/** + * @struct BadDataNeighborOrientationCheckInputValues + * @brief Holds user-facing parameters for the Bad Data Neighbor Orientation Check algorithm. + * + * This filter rehabilitates "bad" voxels (mask == false) by checking whether enough + * of their 6 face-neighbors are "good" and have a similar crystallographic orientation. + * If a bad voxel has at least NumberOfNeighbors good neighbors whose misorientation is + * within MisorientationTolerance, its mask is flipped to true. + */ struct ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckInputValues { - float32 MisorientationTolerance; - int32 NumberOfNeighbors; - DataPath ImageGeomPath; - DataPath QuatsArrayPath; - DataPath MaskArrayPath; - DataPath CellPhasesArrayPath; - DataPath CrystalStructuresArrayPath; + float32 MisorientationTolerance; ///< Maximum allowed misorientation (degrees) between a bad voxel and a good neighbor for the neighbor to count as "matching". + int32 NumberOfNeighbors; ///< Minimum number of matching good face-neighbors required to flip a bad voxel to good. The algorithm iterates from 6 down to this value. + DataPath ImageGeomPath; ///< Path to the ImageGeometry that defines the voxel grid dimensions. + DataPath QuatsArrayPath; ///< Path to the Float32 quaternion array (4 components per tuple) for each voxel. + DataPath MaskArrayPath; ///< Path to the boolean or uint8 mask array (true = good, false = bad). Modified in-place. + DataPath CellPhasesArrayPath; ///< Path to the Int32 array of per-voxel phase IDs. + DataPath CrystalStructuresArrayPath; ///< Path to the UInt32 ensemble array mapping phase ID -> EbsdLib crystal structure enum. }; /** - * @class + * @class BadDataNeighborOrientationCheck + * @brief Dispatcher that selects between in-core and out-of-core neighbor orientation check algorithms. + * + * This class serves as the entry point called by BadDataNeighborOrientationCheckFilter::executeImpl(). + * It inspects the backing storage of the quaternion, mask, and phase arrays using + * DispatchAlgorithm: + * + * - **In-core (BadDataNeighborOrientationCheckWorklist)**: Precomputes a per-voxel neighbor + * count, then uses a deque-based worklist for O(flipped) propagation. Efficient when all + * data is in contiguous RAM because random-access updates to any voxel are O(1). + * + * - **Out-of-core (BadDataNeighborOrientationCheckScanline)**: Uses a 3-slice rolling window + * (prev/cur/next Z-slices) loaded via copyIntoBuffer()/copyFromBuffer(). Neighbor counts + * are recomputed on-the-fly for each bad voxel instead of being stored in a global array, + * because maintaining a global count array would require random-access OOC writes. + * + * @see BadDataNeighborOrientationCheckWorklist, BadDataNeighborOrientationCheckScanline, DispatchAlgorithm */ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheck { public: + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all input and output arrays. + * @param mesgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic flag checked periodically to support user cancellation. + * @param inputValues Pointer to the parameter struct; must outlive this object. + */ BadDataNeighborOrientationCheck(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, BadDataNeighborOrientationCheckInputValues* inputValues); ~BadDataNeighborOrientationCheck() noexcept; @@ -35,15 +67,24 @@ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheck BadDataNeighborOrientationCheck& operator=(const BadDataNeighborOrientationCheck&) = delete; BadDataNeighborOrientationCheck& operator=(BadDataNeighborOrientationCheck&&) noexcept = delete; + /** + * @brief Dispatches to BadDataNeighborOrientationCheckWorklist (in-core) or + * BadDataNeighborOrientationCheckScanline (OOC) based on storage type. + * @return Result<> with any errors. + */ Result<> operator()(); + /** + * @brief Returns the cancellation flag reference. + * @return const reference to the atomic cancellation flag. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp index f715c2daa7..2d71c000e3 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.cpp @@ -16,8 +16,27 @@ using namespace nx::core; namespace { /** - * @brief Computes misorientation between quat1 and the neighbor at neighborSliceIdx. - * Returns true if same phase, phase > 0, and misorientation < tolerance. + * @brief Checks whether a single face-neighbor has matching orientation. + * + * Given a neighbor's slice-local index within one of the rolling window buffers, + * this function checks: + * 1. Same phase as the target voxel (and phase > 0, i.e., not unindexed). + * 2. Misorientation between the target quaternion (quat1) and the neighbor's + * quaternion is below the tolerance threshold. + * + * The misorientation is computed using the Laue-class-specific symmetry operators + * via LaueOps::calculateMisorientation(), which returns the minimum misorientation + * angle across all symmetrically-equivalent representations. + * + * @param neighborSliceIdx Index of the neighbor within the slice buffer (not a global voxel index). + * @param neighborQuats Quaternion buffer for the slice containing the neighbor (4 components per tuple). + * @param neighborPhases Phase buffer for the slice containing the neighbor. + * @param curPhase Phase ID of the target (bad) voxel. + * @param laueClass Crystal structure enum for the target voxel's phase. + * @param quat1 Quaternion of the target (bad) voxel, already in positive orientation. + * @param misorientationTolerance Maximum allowed misorientation in radians. + * @param orientationOps Vector of all LaueOps instances, indexed by crystal structure enum. + * @return true if the neighbor is same-phase with misorientation below tolerance. */ inline bool isMisorientationMatch(int64 neighborSliceIdx, const std::vector& neighborQuats, const std::vector& neighborPhases, int32 curPhase, uint32 laueClass, const ebsdlib::QuatD& quat1, float32 misorientationTolerance, const std::vector& orientationOps) @@ -35,9 +54,31 @@ inline bool isMisorientationMatch(int64 neighborSliceIdx, const std::vector& prevQuats, const std::vector& curQuats, const std::vector& nextQuats, const std::vector& prevPhases, const std::vector& curPhases, @@ -88,19 +129,34 @@ BadDataNeighborOrientationCheckScanline::~BadDataNeighborOrientationCheckScanlin // ----------------------------------------------------------------------------- /** - * @brief Flips bad voxels to good using Z-slice rolling window bulk I/O. + * @brief OOC-safe bad-voxel flipping using Z-slice rolling window bulk I/O. + * + * This algorithm uses O(3 * sliceSize) memory for a 3-slice rolling window + * (previous, current, next Z-slices) instead of any global per-voxel arrays. + * Neighbor counts are recomputed on-the-fly for each bad voxel on every pass, + * trading computation for strictly sequential I/O that avoids chunk thrashing. + * + * **Outer loop** (level from 6 down to NumberOfNeighbors): + * At each level, the required neighbor count to flip a voxel decreases by 1. + * Starting at 6 (all neighbors must agree) and relaxing to the user's threshold + * ensures that high-confidence flips happen first, which in turn enables + * additional flips in subsequent levels (cascade effect). * - * Zero O(N) memory: no per-voxel neighborCount or mask arrays. Instead, - * neighbor counts are recomputed on-the-fly for each bad voxel during every - * pass using only O(slice) rolling window buffers. The mask is read/written - * per-slice from the real OOC-backed mask array. + * **Inner loop** (pass until convergence): + * For each pass at a given level: + * 1. Load Z-slices 0 and 1 into the rolling window. + * 2. Scan every voxel in the current slice. For each bad voxel, recompute the + * count of matching good face-neighbors using the 3-slice window. + * 3. If count >= currentLevel, flip the voxel's mask in the local buffer. + * 4. If any flips occurred in the slice, write the updated mask back to the + * OOC store and set the "changed" flag to trigger another pass. + * 5. Shift the rolling window forward by one Z-slice. * - * For each level (6 down to NumberOfNeighbors), repeatedly scan the volume. - * For each bad voxel, recompute the count of matching good face-neighbors. - * If count >= currentLevel, flip the voxel. Repeat until no flips occur. + * Passes repeat until no voxels flip in a full volume scan, then the level decrements. */ Result<> BadDataNeighborOrientationCheckScanline::operator()() { + // Convert misorientation tolerance from degrees to radians for LaueOps comparison. const float32 misorientationTolerance = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; const auto* imageGeomPtr = m_DataStructure.getDataAs(m_InputValues->ImageGeomPath); @@ -109,6 +165,7 @@ Result<> BadDataNeighborOrientationCheckScanline::operator()() auto& quats = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath); const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); + // Instantiate the mask comparison utility, which handles both bool and uint8 mask types. std::unique_ptr maskCompare; try { @@ -123,11 +180,13 @@ Result<> BadDataNeighborOrientationCheckScanline::operator()() const int64 dimZ = static_cast(udims[2]); const int64 xyStride = dimX * dimY; const usize sliceSize = static_cast(dimY) * static_cast(dimX); - const usize quatSliceElems = sliceSize * 4; + const usize quatSliceElems = sliceSize * 4; // 4 quaternion components per voxel std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); - // Cache ensemble-level CrystalStructures locally (tiny array) + // Cache the ensemble-level crystal structures array locally. This tiny array + // (one entry per phase) is accessed for every neighbor comparison, so caching + // avoids repeated per-element OOC reads. const usize numCrystalStructures = crystalStructures.getNumberOfTuples(); std::vector localCrystalStructures(numCrystalStructures); { @@ -135,10 +194,14 @@ Result<> BadDataNeighborOrientationCheckScanline::operator()() csStore.copyIntoBuffer(0, nonstd::span(localCrystalStructures.data(), numCrystalStructures)); } + // Obtain DataStore references for bulk slice I/O. All per-element access goes + // through copyIntoBuffer()/copyFromBuffer() to maintain sequential access patterns. auto& quatsStore = quats.getDataStoreRef(); const auto& phasesStore = cellPhases.getDataStoreRef(); - // Get the mask DataStore for bulk slice reads and per-element flip writes + // For uint8 masks, we can use bulk copyIntoBuffer()/copyFromBuffer() directly + // on the mask store. For bool masks, we fall back to per-element maskCompare + // access because there is no typed BoolAbstractDataStore with bulk I/O support. auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); const bool maskIsUInt8 = (maskArray.getDataType() == DataType::uint8); AbstractDataStore* maskStorePtr = nullptr; @@ -147,7 +210,10 @@ Result<> BadDataNeighborOrientationCheckScanline::operator()() maskStorePtr = &dynamic_cast(maskArray).getDataStoreRef(); } - // Rolling window buffers — O(slice) memory only + // ---- Rolling window buffers ---- + // Three Z-slices of quaternions, phases, and mask values. These are swapped + // (not copied) as the window advances, so the total memory is 3 * sliceSize + // per array type. std::vector prevQuats(quatSliceElems); std::vector curQuats(quatSliceElems); std::vector nextQuats(quatSliceElems); @@ -158,7 +224,10 @@ Result<> BadDataNeighborOrientationCheckScanline::operator()() std::vector curMask(sliceSize); std::vector nextMask(sliceSize); - // Helper to load a mask slice from the store + // Helper to load a mask slice from the store. For uint8 masks, uses bulk + // copyIntoBuffer() which is efficient for OOC stores. For bool masks, falls + // back to per-element access via maskCompare because BoolArray lacks typed + // bulk I/O support. The bool path is slower but only used in rare cases. auto loadMaskSlice = [&](usize offset, std::vector& dest) { if(maskStorePtr != nullptr) { @@ -174,7 +243,9 @@ Result<> BadDataNeighborOrientationCheckScanline::operator()() } }; - // Helper to load all 3 arrays for a given Z-slice into dest buffers + // Helper to bulk-load all three per-voxel arrays (quaternions, phases, mask) for a + // single Z-slice into the destination buffers. Each call issues 3 copyIntoBuffer() + // calls against the OOC stores, loading one complete Z-slice per array. auto loadSlice = [&](int64 z, std::vector& dstQuats, std::vector& dstPhases, std::vector& dstMask) { const usize offset = static_cast(z) * sliceSize; quatsStore.copyIntoBuffer(offset * 4, nonstd::span(dstQuats.data(), quatSliceElems)); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp index 1057ab1ab4..65f7eade7c 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckScanline.hpp @@ -11,9 +11,52 @@ namespace nx::core struct BadDataNeighborOrientationCheckInputValues; +/** + * @class BadDataNeighborOrientationCheckScanline + * @brief Out-of-core (Scanline) algorithm for the bad-data neighbor orientation check. + * + * This algorithm is selected by the dispatcher when any of the quaternion, mask, or + * phase arrays are backed by chunked (OOC) storage. It avoids random-access patterns + * that would cause chunk thrashing by using a 3-slice rolling window over the Z axis. + * + * **Strategy -- rolling window with on-the-fly neighbor counting**: + * + * For each "level" (starting at 6, decrementing to NumberOfNeighbors), the algorithm + * repeatedly scans the entire volume until no more voxels are flipped: + * + * 1. Load Z-slices 0 (current) and 1 (next) via bulk copyIntoBuffer() for + * quaternions, phases, and the mask. + * 2. For each Z-slice, iterate over every (x, y) in the slice: + * - If the voxel is already good, skip it. + * - Otherwise, check its 6 face-neighbors (4 in-plane from curSlice, 1 from + * prevSlice, 1 from nextSlice) for good voxels with matching orientation. + * - If the count of matching neighbors >= currentLevel, flip the mask to true + * in the local buffer. + * 3. If any voxels were flipped in the current Z-slice, write the updated mask + * back to the OOC store via copyFromBuffer(). + * 4. Shift the rolling window: prev <- cur, cur <- next, and load the next + * Z-slice into the "next" buffer. + * + * **Key difference from the Worklist variant**: Neighbor counts are recomputed from + * scratch for every bad voxel on every pass, because maintaining a persistent global + * neighborCount array would require random-access OOC writes whenever a voxel flips. + * The rolling-window scan approach trades more computation for strictly sequential I/O. + * + * **Memory footprint**: O(3 * sliceSize) for the rolling window buffers -- three + * Z-slices of quaternions, phases, and mask data. No global per-voxel arrays. + * + * @see BadDataNeighborOrientationCheckWorklist for the in-core worklist variant. + */ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckScanline { public: + /** + * @brief Constructs the OOC scanline neighbor orientation check algorithm. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic cancellation flag checked once per Z-slice. + * @param inputValues Pointer to the shared parameter struct; must outlive this object. + */ BadDataNeighborOrientationCheckScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const BadDataNeighborOrientationCheckInputValues* inputValues); ~BadDataNeighborOrientationCheckScanline() noexcept; @@ -23,13 +66,18 @@ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckScanline BadDataNeighborOrientationCheckScanline& operator=(const BadDataNeighborOrientationCheckScanline&) = delete; BadDataNeighborOrientationCheckScanline& operator=(BadDataNeighborOrientationCheckScanline&&) noexcept = delete; + /** + * @brief Flips bad voxels to good using Z-slice rolling window bulk I/O with + * on-the-fly neighbor count recomputation. + * @return Result<> with any errors (e.g., invalid mask path). + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp index c82e84904b..d982d78789 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.cpp @@ -29,13 +29,43 @@ BadDataNeighborOrientationCheckWorklist::~BadDataNeighborOrientationCheckWorklis // ----------------------------------------------------------------------------- /** - * @brief Flips bad voxels to good using a worklist-based propagation algorithm. - * In-core path: Phase 1 counts matching good neighbors per bad voxel. - * Phase 2 iteratively flips eligible voxels using a deque worklist, - * propagating new eligibility to neighbors immediately. + * @brief In-core bad-voxel flipping using two-phase worklist propagation. + * + * This algorithm exploits random-access O(1) DataArray subscript access (safe for + * in-core stores) to achieve O(flipped) amortized cost instead of the O(N * passes) + * cost of the Scanline variant's full-volume rescans. + * + * **Phase 1 -- Initial neighbor counting** (single linear scan, O(N)): + * For every bad voxel, iterate over its 6 face-neighbors. For each good neighbor + * with the same phase and misorientation within tolerance, increment the voxel's + * neighborCount. This produces a baseline count before any flips occur. + * + * **Phase 2 -- Worklist-driven propagation** (per level, O(flipped)): + * For each level (6 down to NumberOfNeighbors): + * 1. Seed a deque with all bad voxels whose neighborCount >= currentLevel. + * 2. Pop the front voxel. If it has already been flipped (by a neighbor cascade) + * or its count has dropped below the threshold (impossible in practice but + * checked defensively), skip it. + * 3. Flip the voxel's mask to true. + * 4. For each still-bad face-neighbor of the newly-flipped voxel: check if the + * neighbor has a matching orientation (same phase, misorientation < tolerance). + * If so, increment its neighborCount. If the count now meets the threshold, + * enqueue the neighbor for processing. + * 5. Repeat until the deque drains, then move to the next level. + * + * This is essentially a breadth-first flood-fill constrained by crystallographic + * misorientation. The cascade effect means that flipping one voxel can immediately + * enable its neighbors to flip, propagating outward from high-confidence seeds. + * + * **Why this is not suitable for OOC**: The deque pops voxels in arbitrary spatial + * order (BFS wavefront). Each pop accesses the popped voxel's quaternion, phase, + * and mask, plus all 6 neighbors' data -- all random-access lookups. On OOC stores, + * each such lookup could trigger a disk-chunk load/evict, creating catastrophic + * chunk thrashing for large datasets. */ Result<> BadDataNeighborOrientationCheckWorklist::operator()() { + // Convert misorientation tolerance from degrees to radians. const float32 misorientationTolerance = m_InputValues->MisorientationTolerance * numbers::pi_v / 180.0f; const auto* imageGeomPtr = m_DataStructure.getDataAs(m_InputValues->ImageGeomPath); @@ -62,22 +92,32 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() const int64 xyStride = dims[0] * dims[1]; + // Precompute the 6 face-neighbor index offsets (-X, +X, -Y, +Y, -Z, +Z) relative + // to a voxel's linear index in the flat array. These are constant for any given + // volume geometry. std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + // Per-voxel count of good face-neighbors with matching orientation. This O(N) array + // is the trade-off: the Worklist variant uses O(N) memory to achieve O(flipped) + // propagation speed, while the Scanline variant uses O(slice) memory but O(N * passes). std::vector neighborCount(totalPoints, 0); // ===== Phase 1: Count matching good neighbors for each bad voxel ===== + // Single linear scan over all voxels. For each bad voxel, check its 6 face-neighbors + // for good voxels with matching phase and orientation within tolerance. for(usize voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) { if(!maskCompare->isTrue(voxelIndex)) { + // Build the target voxel's quaternion for misorientation comparisons. ebsdlib::QuatD quat1(quats[voxelIndex * 4], quats[voxelIndex * 4 + 1], quats[voxelIndex * 4 + 2], quats[voxelIndex * 4 + 3]); quat1.positiveOrientation(); const uint32 laueClass1 = crystalStructures[cellPhases[voxelIndex]]; + // Decompose the linear index into (x, y, z) coordinates for boundary checks. const int64 xIdx = static_cast(voxelIndex) % dims[0]; const int64 yIdx = (static_cast(voxelIndex) / dims[0]) % dims[1]; const int64 zIdx = static_cast(voxelIndex) / xyStride; @@ -91,6 +131,8 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() } const int64 neighborPoint = static_cast(voxelIndex) + neighborVoxelIndexOffsets[faceIndex]; + // Only count good neighbors (mask == true) with the same phase and a + // misorientation below the tolerance. if(maskCompare->isTrue(neighborPoint)) { if(cellPhases[voxelIndex] == cellPhases[neighborPoint] && cellPhases[voxelIndex] > 0) @@ -109,11 +151,15 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() } // ===== Phase 2: Iteratively flip bad voxels using worklist ===== + // Iterate from the strictest level (6 = all face-neighbors must agree) down to + // the user's minimum. At each level, seed the worklist with all eligible voxels, + // then drain it with propagation. constexpr int32 startLevel = 6; const int32 totalLevels = startLevel - m_InputValues->NumberOfNeighbors + 1; for(int32 currentLevel = startLevel; currentLevel >= m_InputValues->NumberOfNeighbors; currentLevel--) { + // Seed the worklist with all bad voxels that already meet this level's threshold. std::deque worklist; for(usize voxelIndex = 0; voxelIndex < totalPoints; voxelIndex++) { @@ -123,18 +169,25 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() } } + // Process the worklist. When a voxel is flipped, its still-bad neighbors may + // gain a new matching good neighbor and become eligible, creating a cascade. while(!worklist.empty()) { const usize voxelIndex = worklist.front(); worklist.pop_front(); + // Defensive check: skip if already flipped (by a prior cascade) or if the + // count dropped below threshold (should not happen, but guards correctness). if(maskCompare->isTrue(voxelIndex) || neighborCount[voxelIndex] < currentLevel) { continue; } + // Flip this voxel from bad to good. maskCompare->setValue(voxelIndex, true); + // Now propagate: for each still-bad face-neighbor, check if the newly-flipped + // voxel constitutes a new matching good neighbor for that neighbor. ebsdlib::QuatD quat1(quats[voxelIndex * 4], quats[voxelIndex * 4 + 1], quats[voxelIndex * 4 + 2], quats[voxelIndex * 4 + 3]); quat1.positiveOrientation(); const uint32 laueClass1 = crystalStructures[cellPhases[voxelIndex]]; @@ -162,7 +215,12 @@ Result<> BadDataNeighborOrientationCheckWorklist::operator()() ebsdlib::AxisAngleDType axisAngle = orientationOps[laueClass1]->calculateMisorientation(quat1, quat2); if(axisAngle[3] < misorientationTolerance) { + // Increment the neighbor's count because the just-flipped voxel is + // now a new good neighbor for it. neighborCount[neighborPoint]++; + // If the neighbor now meets the threshold, enqueue it for processing. + // It may be enqueued multiple times as different neighbors flip, but + // the defensive check at the top of the while loop handles duplicates. if(neighborCount[neighborPoint] >= currentLevel) { worklist.push_back(static_cast(neighborPoint)); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp index 5bbba4e92c..01f03ff55e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/BadDataNeighborOrientationCheckWorklist.hpp @@ -11,9 +11,51 @@ namespace nx::core struct BadDataNeighborOrientationCheckInputValues; +/** + * @class BadDataNeighborOrientationCheckWorklist + * @brief In-core (Worklist) algorithm for the bad-data neighbor orientation check. + * + * This algorithm is selected by the dispatcher when all relevant arrays reside in + * contiguous in-memory DataStores. It operates in two phases: + * + * **Phase 1 -- Initial neighbor counting** (single linear scan): + * For every bad voxel, count how many of its 6 face-neighbors are good and have + * a crystallographic misorientation within the tolerance. Store this count in a + * per-voxel neighborCount[N] array. + * + * **Phase 2 -- Worklist-driven propagation** (per level, 6 down to NumberOfNeighbors): + * 1. Seed a deque with all bad voxels whose neighborCount >= currentLevel. + * 2. Pop the front voxel. If it is still bad and still eligible, flip its mask + * to true. + * 3. For each still-bad face-neighbor of the newly-flipped voxel: if the neighbor + * has matching orientation (same phase, misorientation < tolerance), increment + * its neighborCount. If the count now meets the threshold, enqueue the neighbor. + * 4. Repeat until the deque is empty, then move to the next level. + * + * This worklist approach has O(flipped) amortized cost because each voxel is + * processed at most once per level, and neighbors are only re-examined when a + * neighboring voxel actually flips. In contrast, the Scanline variant must re-scan + * the entire volume on every pass. + * + * **Memory footprint**: O(N) for the neighborCount array (one int32 per voxel) plus + * O(worklist size) for the deque. + * + * **Why this is not suitable for OOC**: The random-access pattern (deque pops voxels + * in arbitrary order, then accesses their neighbors) would trigger catastrophic chunk + * thrashing on disk-backed stores. + * + * @see BadDataNeighborOrientationCheckScanline for the OOC-optimized variant. + */ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckWorklist { public: + /** + * @brief Constructs the in-core worklist neighbor orientation check algorithm. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic cancellation flag. + * @param inputValues Pointer to the shared parameter struct; must outlive this object. + */ BadDataNeighborOrientationCheckWorklist(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const BadDataNeighborOrientationCheckInputValues* inputValues); ~BadDataNeighborOrientationCheckWorklist() noexcept; @@ -23,13 +65,17 @@ class ORIENTATIONANALYSIS_EXPORT BadDataNeighborOrientationCheckWorklist BadDataNeighborOrientationCheckWorklist& operator=(const BadDataNeighborOrientationCheckWorklist&) = delete; BadDataNeighborOrientationCheckWorklist& operator=(BadDataNeighborOrientationCheckWorklist&&) noexcept = delete; + /** + * @brief Flips bad voxels to good using two-phase worklist propagation. + * @return Result<> with any errors (e.g., invalid mask path). + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const BadDataNeighborOrientationCheckInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp index 343a16301e..20228c34a4 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/CAxisSegmentFeatures.hpp @@ -11,30 +11,74 @@ namespace nx::core { +/** + * @struct CAxisSegmentFeaturesInputValues + * @brief Holds all user-supplied parameters for the CAxisSegmentFeatures algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT CAxisSegmentFeaturesInputValues { - float32 MisorientationTolerance; - bool UseMask; - bool RandomizeFeatureIds; - SegmentFeatures::NeighborScheme NeighborScheme; - DataPath ImageGeometryPath; - DataPath QuatsArrayPath; - DataPath CellPhasesArrayPath; - DataPath MaskArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath FeatureIdsArrayPath; - DataPath CellFeatureAttributeMatrixPath; - DataPath ActiveArrayPath; + float32 MisorientationTolerance = 0.0f; ///< Maximum c-axis misalignment angle (radians) for grouping voxels into the same feature + bool UseMask = false; ///< Whether to exclude masked voxels from segmentation + bool RandomizeFeatureIds = false; ///< Whether to randomize feature IDs after segmentation for better visual contrast + SegmentFeatures::NeighborScheme NeighborScheme{}; ///< Face-only (6) or all-connected (26) neighbor connectivity + DataPath ImageGeometryPath; ///< Path to the ImageGeom defining the 3D voxel grid + DataPath QuatsArrayPath; ///< Path to the Float32 quaternion array (4 components per cell) + DataPath CellPhasesArrayPath; ///< Path to the Int32 cell phases array + DataPath MaskArrayPath; ///< Path to the Bool/UInt8 mask array (only used when UseMask is true) + DataPath CrystalStructuresArrayPath; ///< Path to the UInt32 crystal structures ensemble array + DataPath FeatureIdsArrayPath; ///< Path to the output Int32 feature IDs array + DataPath CellFeatureAttributeMatrixPath; ///< Path to the Feature-level AttributeMatrix (resized to match feature count) + DataPath ActiveArrayPath; ///< Path to the output UInt8 Active array (1 = active feature, 0 = reserved slot 0) }; /** * @class CAxisSegmentFeatures - * @brief This filter segments the Features by grouping neighboring Cells that satisfy the C-axis misalignment tolerance, i.e., have misalignment angle less than the value set by the user. + * @brief Segments a hexagonal EBSD dataset into features (grains) based on + * c-axis alignment rather than full crystallographic misorientation. + * + * The c-axis is the [0001] direction in hexagonal crystal systems (HCP metals + * like titanium, zirconium, magnesium). Two neighboring voxels are grouped into + * the same feature when the angle between their sample-frame c-axis directions + * is within MisorientationTolerance. Because the c-axis is bidirectional + * (parallel and antiparallel are equivalent), the check accepts both + * w <= tolerance and (pi - w) <= tolerance. + * + * The c-axis direction is obtained by converting each voxel's quaternion + * orientation to a 3x3 orientation matrix, transposing it, and multiplying by + * the crystal-frame c-axis unit vector [0,0,1] to get the sample-frame direction. + * + * This filter requires all phases to be hexagonal (Hexagonal_High 6/mmm or + * Hexagonal_Low 6/m). A pre-validation pass checks every cell's phase; if + * any non-hexagonal phase is found, the filter returns an error. + * + * ## Algorithm Dispatch + * + * Identical to EBSDSegmentFeatures: dispatches between DFS flood fill (in-core) + * and connected-component labeling (OOC) based on IsOutOfCore() or + * ForceOocAlgorithm(). + * + * ## Rolling 2-Slot Slice Buffers (OOC Optimization) + * + * Same architecture as EBSDSegmentFeatures: prepareForSlice() bulk-reads each + * Z-slice's quaternion, phase, and mask data into a rolling 2-slot buffer. + * The isValidVoxel() and areNeighborsSimilar() methods use the buffer fast path + * when the needed slices are loaded, falling back to direct DataStore access + * for non-adjacent slice comparisons during periodic boundary merging. + * + * The pre-validation phase scan also uses OOC-efficient batch reading: phases + * are read one Z-slice at a time via copyIntoBuffer() rather than per-element + * getValue() calls. */ - class ORIENTATIONANALYSIS_EXPORT CAxisSegmentFeatures : public SegmentFeatures { public: + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Handler for sending progress messages to the UI. + * @param shouldCancel Atomic flag checked between iterations to support cancellation. + * @param inputValues User-supplied parameters controlling the segmentation behavior. + */ CAxisSegmentFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, CAxisSegmentFeaturesInputValues* inputValues); ~CAxisSegmentFeatures() noexcept override; @@ -43,47 +87,103 @@ class ORIENTATIONANALYSIS_EXPORT CAxisSegmentFeatures : public SegmentFeatures CAxisSegmentFeatures& operator=(const CAxisSegmentFeatures&) = delete; CAxisSegmentFeatures& operator=(CAxisSegmentFeatures&&) noexcept = delete; + /** + * @brief Executes the c-axis segmentation algorithm, dispatching between DFS + * (in-core) and CCL (OOC) strategies based on the backing DataStore type. + * @return Result<> indicating success or any errors encountered during execution. + */ Result<> operator()(); protected: + /** + * @brief Finds the next unassigned voxel to serve as a seed for a new feature. + * Used by the DFS flood-fill path (execute()). + * @param gnum The feature number to assign to the seed voxel. + * @param nextSeed Linear index to start scanning from. + * @return Linear index of the seed voxel, or -1 if no more seeds exist. + */ int64 getSeed(int32 gnum, int64 nextSeed) const override; + + /** + * @brief Determines whether a neighbor should be merged into the current + * feature during DFS flood fill. Checks mask, phase equality, and + * c-axis alignment (not full misorientation). + * @param referencePoint Linear index of the current (reference) voxel. + * @param neighborPoint Linear index of the candidate neighbor voxel. + * @param gnum The feature number to assign if the neighbor is accepted. + * @return true if the neighbor was merged (its featureId set to gnum). + */ bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; /** - * @brief Checks whether a voxel can participate in C-axis segmentation based on mask and phase. + * @brief Checks whether a voxel can participate in c-axis segmentation. + * + * Used by the CCL path. When slice buffers are active, reads from the + * in-memory buffer (fast path); otherwise falls back to direct DataStore + * access (OOC fallback). + * * @param point Linear voxel index. - * @return true if the voxel passes mask and phase checks. + * @return true if the voxel passes mask and phase > 0 checks. */ bool isValidVoxel(int64 point) const override; /** - * @brief Determines whether two neighboring voxels belong to the same C-axis segment. + * @brief Determines whether two neighboring voxels have sufficiently aligned + * c-axes to belong to the same feature. Used by the CCL path. + * + * When both voxels' Z-slices are in the rolling buffer, all data is read + * from local memory. Otherwise falls back to direct DataStore access. + * * @param point1 First voxel index. * @param point2 Second (neighbor) voxel index. - * @return true if both voxels share the same phase and their C-axis misalignment is within tolerance. + * @return true if both share the same phase and their c-axis misalignment is within tolerance. */ bool areNeighborsSimilar(int64 point1, int64 point2) const override; + /** + * @brief Pre-loads a Z-slice's quaternion, phase, and mask data into the + * rolling 2-slot buffer for OOC-efficient access during CCL. + * + * Slot assignment: even slices -> slot 0, odd slices -> slot 1. Passing + * iz < 0 disables slice buffering. + * + * @param iz Z-slice index to load, or -1 to disable buffering. + * @param dimX Number of voxels in X. + * @param dimY Number of voxels in Y. + * @param dimZ Number of voxels in Z. + */ void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) override; private: - const CAxisSegmentFeaturesInputValues* m_InputValues = nullptr; + const CAxisSegmentFeaturesInputValues* m_InputValues = nullptr; ///< Non-owning pointer to user-supplied parameters - Float32Array* m_QuatsArray = nullptr; - Int32Array* m_CellPhases = nullptr; - std::unique_ptr m_GoodVoxelsArray = nullptr; - Int32Array* m_FeatureIdsArray = nullptr; + Float32Array* m_QuatsArray = nullptr; ///< Pointer to the quaternion array (4 components per cell) + Int32Array* m_CellPhases = nullptr; ///< Pointer to the cell phases array + std::unique_ptr m_GoodVoxelsArray = nullptr; ///< Mask comparator for filtering valid voxels + Int32Array* m_FeatureIdsArray = nullptr; ///< Pointer to the output feature IDs array + /** + * @brief Allocates the rolling 2-slot slice buffers for OOC optimization. + * Each slot holds one full XY slice of quaternions, phases, and mask flags. + * @param dimX Number of voxels in X. + * @param dimY Number of voxels in Y. + */ void allocateSliceBuffers(int64 dimX, int64 dimY); + + /** + * @brief Releases the slice buffers and resets m_UseSliceBuffers to false. + */ void deallocateSliceBuffers(); // Rolling 2-slot input buffers for OOC optimization. - std::vector m_QuatBuffer; - std::vector m_PhaseBuffer; - std::vector m_MaskBuffer; - int64 m_BufSliceSize = 0; - int64 m_BufferedSliceZ[2] = {-1, -1}; - bool m_UseSliceBuffers = false; + // Pre-loading input data into these avoids per-element OOC overhead + // during neighbor comparisons in the CCL algorithm. + std::vector m_QuatBuffer; ///< 2 * sliceSize * 4 quaternion components + std::vector m_PhaseBuffer; ///< 2 * sliceSize phase IDs + std::vector m_MaskBuffer; ///< 2 * sliceSize mask flags + int64 m_BufSliceSize = 0; ///< Number of voxels per XY slice (dimX * dimY) + int64 m_BufferedSliceZ[2] = {-1, -1}; ///< Z-indices currently loaded in each buffer slot + bool m_UseSliceBuffers = false; ///< Whether slice buffers are active }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp index bb507d05d6..c5cbc0c26f 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.cpp @@ -38,9 +38,23 @@ const std::atomic_bool& ComputeAvgCAxes::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Computes the average crystallographic c-axis direction per feature for + * hexagonal phases. Each cell's quaternion is converted to a passive rotation + * matrix, transposed to an active rotation, and multiplied by <0,0,1> to get + * the c-axis in the sample reference frame. A running average c-axis is + * accumulated per feature, then normalized. + * + * OOC strategy: Cell-level arrays (featureIds, phases, quats) are read in + * fixed-size chunks (k_ChunkSize tuples) via copyIntoBuffer. Feature-level + * avgCAxes is cached entirely in a local buffer (random access by featureId + * would cause severe OOC chunk thrashing). The final result is bulk-written + * back to the DataStore via copyFromBuffer. + */ Result<> ComputeAvgCAxes::operator()() { - // Cache ensemble-level crystal structures locally to avoid per-element OOC overhead + // Bulk-read ensemble-level crystal structures into local memory to avoid + // per-element OOC virtual dispatch during the cell loop const auto& crystalStructuresStoreRef = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); const usize numCrystalStructures = crystalStructuresStoreRef.getSize(); auto crystalStructuresCache = std::make_unique(numCrystalStructures); @@ -71,6 +85,8 @@ Result<> ComputeAvgCAxes::operator()() result.warnings().push_back({-76403, "Non Hexagonal phases were found. All calculations for non Hexagonal phases will be skipped and a NaN value inserted."}); } + // DataStore references for cell-level arrays — all access goes through + // copyIntoBuffer/copyFromBuffer to avoid per-element OOC overhead. const auto& featureIdsStoreRef = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); const auto& quatsStoreRef = m_DataStructure.getDataAs(m_InputValues->QuatsArrayPath)->getDataStoreRef(); const auto& cellPhasesStoreRef = m_DataStructure.getDataAs(m_InputValues->CellPhasesArrayPath)->getDataStoreRef(); @@ -79,7 +95,9 @@ Result<> ComputeAvgCAxes::operator()() const usize totalPoints = featureIdsStoreRef.getNumberOfTuples(); const usize totalFeatures = avgCAxesStoreRef.getNumberOfTuples(); - // Cache feature-level avgCAxes locally (random access by featureId in the hot loop) + // Cache feature-level avgCAxes entirely in a local buffer. The cell loop + // accesses this by featureId (random order), which would cause severe OOC + // chunk thrashing if left in the DataStore. const usize avgCAxesElements = totalFeatures * 3; auto avgCAxesCache = std::make_unique(avgCAxesElements); avgCAxesStoreRef.copyIntoBuffer(0, nonstd::span(avgCAxesCache.get(), avgCAxesElements)); @@ -89,12 +107,14 @@ Result<> ComputeAvgCAxes::operator()() auto counter = std::make_unique(totalFeatures); - // Allocate chunk buffers for cell-level arrays (sequential access) + // Pre-allocate chunk buffers for sequential cell-level reads. These are + // reused every iteration to avoid repeated heap allocations. auto featureIdsChunk = std::make_unique(k_ChunkSize); auto cellPhasesChunk = std::make_unique(k_ChunkSize); auto quatsChunk = std::make_unique(k_ChunkSize * 4); - // Loop over each cell in chunks to minimize OOC overhead + // Process cells in fixed-size chunks. Each chunk triggers one bulk read per + // array, amortizing OOC overhead over k_ChunkSize tuples. usize tupleIdx = 0; while(tupleIdx < totalPoints) { @@ -105,7 +125,7 @@ Result<> ComputeAvgCAxes::operator()() const usize chunkTuples = std::min(k_ChunkSize, totalPoints - tupleIdx); - // Bulk-read cell-level data for this chunk + // Bulk-read this chunk of cell data (sequential access pattern, OOC-friendly) featureIdsStoreRef.copyIntoBuffer(tupleIdx, nonstd::span(featureIdsChunk.get(), chunkTuples)); cellPhasesStoreRef.copyIntoBuffer(tupleIdx, nonstd::span(cellPhasesChunk.get(), chunkTuples)); quatsStoreRef.copyIntoBuffer(tupleIdx * 4, nonstd::span(quatsChunk.get(), chunkTuples * 4)); @@ -200,7 +220,8 @@ Result<> ComputeAvgCAxes::operator()() } } - // Write cached avgCAxes data back to the store + // Single bulk-write of the completed feature-level avgCAxes back to the DataStore. + // All accumulation and normalization was done in the local buffer. avgCAxesStoreRef.copyFromBuffer(0, nonstd::span(avgCAxesCache.get(), avgCAxesElements)); return result; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.hpp index 45cd115ed5..b9e3c07c78 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgCAxes.hpp @@ -9,21 +9,43 @@ namespace nx::core { +/** + * @brief Input values for the ComputeAvgCAxes algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeAvgCAxesInputValues { - DataPath QuatsArrayPath; - DataPath FeatureIdsArrayPath; - DataPath CellPhasesArrayPath; - DataPath CellFeatureDataPath; - DataPath AvgCAxesArrayPath; - DataPath CrystalStructuresArrayPath; + DataPath QuatsArrayPath; ///< Cell-level Float32 quaternions (4 components) + DataPath FeatureIdsArrayPath; ///< Cell-level Int32 feature ID per voxel + DataPath CellPhasesArrayPath; ///< Cell-level Int32 phase index per voxel + DataPath CellFeatureDataPath; ///< Feature-level AttributeMatrix path + DataPath AvgCAxesArrayPath; ///< Output: Feature-level Float32 average c-axis (3 components) + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes }; /** * @class ComputeAvgCAxes - * @brief This filter determines the average C-axis location of each Feature. + * @brief Computes the average crystallographic c-axis direction for each Feature + * in the sample reference frame. + * + * For each voxel belonging to a Feature, the quaternion is converted to an + * orientation matrix, transposed (passive to active), and multiplied by the + * <001> c-axis direction to obtain the c-axis in the sample frame. A running + * average is maintained per Feature with sign flipping to keep the accumulated + * directions in the same hemisphere. + * + * Only Hexagonal-High (6/mmm) and Hexagonal-Low (6/m) Laue classes are + * supported; non-hexagonal phases produce NaN output values. + * + * ## OOC Optimization + * + * Cell-level arrays (featureIds, phases, quats) are read in chunks of 4096 + * tuples via `copyIntoBuffer()`. Ensemble-level crystal structures and + * feature-level avgCAxes are cached entirely in local `std::vector`s. + * The final averaged result is written back to the DataStore in a single + * `copyFromBuffer()` call. This eliminates per-element virtual dispatch + * overhead that causes severe performance degradation when data is stored + * out-of-core in chunked format. */ - class ORIENTATIONANALYSIS_EXPORT ComputeAvgCAxes { public: @@ -35,6 +57,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeAvgCAxes ComputeAvgCAxes& operator=(const ComputeAvgCAxes&) = delete; ComputeAvgCAxes& operator=(ComputeAvgCAxes&&) noexcept = delete; + /** + * @brief Executes the average c-axis computation using chunked bulk I/O. + * @return Result<> with any errors or warnings (e.g., non-hexagonal phases). + */ Result<> operator()(); const std::atomic_bool& getCancel(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp index 98641b15a3..273171f9c1 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.cpp @@ -367,6 +367,18 @@ Result<> ComputeAvgOrientations::computeVmfWatsonAverage() } // ----------------------------------------------------------------------------- +/** + * @brief Computes the average quaternion orientation for each feature using + * iterative Rodrigues averaging. For each cell, the voxel quaternion is rotated + * to the nearest equivalent of the running average, then accumulated. After all + * cells are processed, the accumulated quaternions are normalized, forced into + * the positive hemisphere, and converted to Euler angles. + * + * OOC strategy: Cell-level arrays (featureIds, phases, quats) are read in + * sequential 64K-tuple chunks via copyIntoBuffer to avoid random OOC page + * faults. Feature-level accumulation uses local std::vector buffers (small + * enough to fit in RAM). Final results are bulk-written back via copyFromBuffer. + */ Result<> ComputeAvgOrientations::computeRodriguesAverage() { std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); @@ -385,22 +397,27 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() const usize totalFeatures = avgQuatsStore.getNumberOfTuples(); std::vector counts(totalFeatures, 0.0f); - // Cache crystal structures locally (ensemble-level, tiny) + // Bulk-read crystal structures into a local vector (ensemble-level, typically < 10 entries). + // This avoids repeated virtual dispatch into the OOC DataStore during the hot cell loop. const usize numPhases = crystalStructuresArray.getNumberOfTuples(); std::vector crystalStructures(numPhases); crystalStructuresArray.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructures.data(), numPhases)); - // Local cache for avgQuats (feature-level, manageable) + // Feature-level quaternion accumulator — kept entirely in RAM. Random access + // by featureId would thrash OOC chunks if stored in a DataStore directly. std::vector localAvgQuats(totalFeatures * 4, 0.0f); // Get the Identity Quaternion static const ebsdlib::QuatF identityQuat(0.0f, 0.0f, 0.0f, 1.0f); - // Chunked accumulation of cell-level data + // Obtain DataStore references for bulk I/O. The stores may be backed by + // HDF5 chunked storage, so element-wise operator[] would trigger expensive + // page faults. All cell-level reads go through copyIntoBuffer instead. const auto& featureIdsStore = featureIds.getDataStoreRef(); const auto& phasesStore = phases.getDataStoreRef(); const auto& quatsStore = quats.getDataStoreRef(); + // Pre-allocate chunk buffers (reused across iterations to avoid allocation churn) auto featureIdBuf = std::make_unique(k_ChunkTuples); auto phasesBuf = std::make_unique(k_ChunkTuples); auto quatsBuf = std::make_unique(k_ChunkTuples * 4); @@ -413,6 +430,8 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() } const usize count = std::min(k_ChunkTuples, totalPoints - offset); + // Sequential bulk reads — each call fetches one contiguous chunk from the + // underlying DataStore (a single HDF5 read or memcpy for in-core stores). featureIdsStore.copyIntoBuffer(offset, nonstd::span(featureIdBuf.get(), count)); phasesStore.copyIntoBuffer(offset, nonstd::span(phasesBuf.get(), count)); quatsStore.copyIntoBuffer(offset * 4, nonstd::span(quatsBuf.get(), count * 4)); @@ -421,27 +440,33 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() { const int32 currentFeatureId = featureIdBuf[i]; const int32 currentPhase = phasesBuf[i]; + // Phase index must be > 0 (index 0 is reserved for "unknown" in CrystalStructures). if(currentPhase > 0) { - const uint32 xtal = crystalStructures[currentPhase]; + const uint32 xtal = crystalStructures[currentPhase]; // Laue class from local cache counts[currentFeatureId] += 1.0f; + // Read the voxel quaternion from the chunk buffer (not from DataStore) const usize qi = i * 4; ebsdlib::QuatF voxQuat(quatsBuf[qi], quatsBuf[qi + 1], quatsBuf[qi + 2], quatsBuf[qi + 3]); + // Read the running average from the local accumulator (random access by featureId) const usize fi = static_cast(currentFeatureId) * 4; ebsdlib::QuatF curAvgQuat(localAvgQuats[fi], localAvgQuats[fi + 1], localAvgQuats[fi + 2], localAvgQuats[fi + 3]); ebsdlib::QuatF finalAvgQuat = curAvgQuat; curAvgQuat = curAvgQuat.scalarDivide(counts[currentFeatureId]); + // First voxel: seed with identity so getNearestQuat has a valid reference if(counts[currentFeatureId] == 1.0f) { curAvgQuat = ebsdlib::QuatF::identity(); } + // Rotate voxQuat to the symmetrically equivalent orientation nearest the running average voxQuat = orientationOps[xtal]->getNearestQuat(curAvgQuat, voxQuat); curAvgQuat = finalAvgQuat + voxQuat; + // Write back into local accumulator (not into DataStore — avoids OOC writes) localAvgQuats[fi] = curAvgQuat.x(); localAvgQuats[fi + 1] = curAvgQuat.y(); localAvgQuats[fi + 2] = curAvgQuat.z(); @@ -450,7 +475,8 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() } } - // Second pass: normalize and convert to Euler angles (feature-level only) + // Second pass: normalize accumulated quaternions and convert to Euler angles. + // This is feature-level only (O(features)), so no chunking needed. std::vector localAvgEuler(totalFeatures * 3, 0.0f); for(usize featureId = 0; featureId < totalFeatures; featureId++) @@ -485,7 +511,8 @@ Result<> ComputeAvgOrientations::computeRodriguesAverage() localAvgEuler[ei + 2] = eu[2]; } - // Write feature-level results back to DataStore + // Bulk-write feature-level results back to DataStore in a single operation. + // This is the only write to these stores — all accumulation was done in RAM. avgQuatsStore.copyFromBuffer(0, nonstd::span(localAvgQuats.data(), localAvgQuats.size())); avgEulerStore.copyFromBuffer(0, nonstd::span(localAvgEuler.data(), localAvgEuler.size())); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.hpp index b78f634067..12578de410 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeAvgOrientations.hpp @@ -14,36 +14,72 @@ namespace nx::core { /** - * @brief The ComputeAvgOrientationsInputValues struct + * @brief Input values for the ComputeAvgOrientations algorithm. + * + * All DataPath members are validated by the filter's preflight. Cell-level + * arrays (featureIds, phases, quats) may contain millions of tuples and are + * accessed through chunked bulk I/O in the optimized Rodrigues path. + * Feature- and ensemble-level arrays are small enough to cache entirely in + * local memory. */ struct ORIENTATIONANALYSIS_EXPORT ComputeAvgOrientationsInputValues { - DataPath cellFeatureIdsArrayPath; - DataPath cellPhasesArrayPath; - DataPath cellQuatsArrayPath; - DataPath crystalStructuresArrayPath; - - bool useRodriguesAverage; - bool useVonMisesAverage; - bool useWatsonAverage; - DataPath avgQuatsArrayPath; - DataPath avgEulerAnglesArrayPath; - - DataPath VMFQuatsArrayPath; - DataPath VMFEulerAnglesArrayPath; - DataPath VMFKappaArrayPath; - - DataPath WatsonQuatsArrayPath; - DataPath WatsonEulerAnglesArrayPath; - DataPath WatsonKappaArrayPath; - - uint32 RandomSeed = 43514; - int32 NumEMIterations = 5; - int32 NumIterations = 10; + DataPath cellFeatureIdsArrayPath; ///< Cell-level Int32 array mapping each voxel to its feature + DataPath cellPhasesArrayPath; ///< Cell-level Int32 array of phase indices + DataPath cellQuatsArrayPath; ///< Cell-level Float32 array of quaternions (4 components) + DataPath crystalStructuresArrayPath; ///< Ensemble-level UInt32 array of crystal structure Laue classes + + bool useRodriguesAverage; ///< Enable the Rodrigues (running-sum) averaging method + bool useVonMisesAverage; ///< Enable the von Mises-Fisher EM averaging method + bool useWatsonAverage; ///< Enable the Watson EM averaging method + DataPath avgQuatsArrayPath; ///< Output: Rodrigues average quaternions (feature-level) + DataPath avgEulerAnglesArrayPath; ///< Output: Rodrigues average Euler angles (feature-level) + + DataPath VMFQuatsArrayPath; ///< Output: vMF average quaternions (feature-level) + DataPath VMFEulerAnglesArrayPath; ///< Output: vMF average Euler angles (feature-level) + DataPath VMFKappaArrayPath; ///< Output: vMF kappa concentration values (feature-level) + + DataPath WatsonQuatsArrayPath; ///< Output: Watson average quaternions (feature-level) + DataPath WatsonEulerAnglesArrayPath; ///< Output: Watson average Euler angles (feature-level) + DataPath WatsonKappaArrayPath; ///< Output: Watson kappa concentration values (feature-level) + + uint32 RandomSeed = 43514; ///< Fixed seed for EM reproducibility + int32 NumEMIterations = 5; ///< Number of outer EM iterations + int32 NumIterations = 10; ///< Number of inner iterations per EM cycle }; /** - * @brief + * @class ComputeAvgOrientations + * @brief Computes the average orientation of each Feature from its constituent + * voxel quaternions, using one or more averaging methods. + * + * Three independent methods are available (each can be toggled on/off): + * 1. **Rodrigues average** -- running quaternion sum with nearest-quat selection + * 2. **Von Mises-Fisher (vMF) average** -- EM-based estimation on the unit quaternion sphere + * 3. **Watson average** -- EM-based estimation with antipodal symmetry + * + * ## OOC Optimization (Rodrigues path) + * + * The Rodrigues path iterates over every voxel to accumulate quaternion sums + * per feature. In the original implementation each voxel accessed cell-level + * DataArrays via `operator[]`, which triggers a virtual dispatch per element. + * When the DataStore is backed by an out-of-core (OOC) chunked store this + * causes a chunk load/evict cycle on every access, making the algorithm + * orders of magnitude slower. + * + * The optimized implementation: + * - Caches ensemble-level crystal structures into a local `std::vector` + * (tiny -- one entry per phase). + * - Accumulates running quaternion averages in a local `std::vector` + * (feature-level -- one quaternion per feature, manageable). + * - Reads cell-level arrays (featureIds, phases, quats) in fixed-size + * chunks of 65536 tuples via `copyIntoBuffer()`, processing each chunk + * from contiguous local memory. + * - Writes feature-level results (avgQuats, avgEuler) back to the + * DataStore in a single `copyFromBuffer()` call. + * + * This converts O(N) random virtual dispatches into O(N/chunk) bulk I/O + * operations, eliminating the OOC performance cliff. */ class ORIENTATIONANALYSIS_EXPORT ComputeAvgOrientations { @@ -56,8 +92,16 @@ class ORIENTATIONANALYSIS_EXPORT ComputeAvgOrientations ComputeAvgOrientations& operator=(const ComputeAvgOrientations&) = delete; // Copy Assignment Not Implemented ComputeAvgOrientations& operator=(ComputeAvgOrientations&&) = delete; // Move Assignment Not Implemented + /** + * @brief Executes the enabled averaging methods and populates the output arrays. + * @return Result<> with any errors or warnings encountered during execution. + */ Result<> operator()(); + /** + * @brief Thread-safe progress message emitter used by the vMF/Watson parallel path. + * @param counter Number of features processed since last call. + */ void sendThreadSafeProgressMessage(usize counter); protected: @@ -67,7 +111,18 @@ class ORIENTATIONANALYSIS_EXPORT ComputeAvgOrientations const std::atomic_bool& m_ShouldCancel; const ComputeAvgOrientationsInputValues* m_InputValues = nullptr; + /** + * @brief Computes the Rodrigues (running-sum) average orientation per feature. + * + * Uses chunked bulk I/O for cell-level arrays and local buffers for + * feature-level accumulation to avoid per-element OOC overhead. + */ Result<> computeRodriguesAverage(); + + /** + * @brief Computes von Mises-Fisher and/or Watson average orientations per feature + * using an Expectation-Maximization algorithm. + */ Result<> computeVmfWatsonAverage(); // Thread safe Progress Message diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp index 083ecbdac7..c2515a1603 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.cpp @@ -31,6 +31,17 @@ const std::atomic_bool& ComputeCAxisLocations::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Converts each cell's quaternion to a c-axis direction in the sample + * reference frame. Only hexagonal phases (6/m, 6/mmm) are processed; all + * others receive NaN output values. + * + * OOC strategy: Cell-level arrays (quaternions, phases, output) are processed + * in fixed-size chunks (65K tuples). Each chunk is bulk-read via + * copyIntoBuffer, processed locally, and the output is bulk-written via + * copyFromBuffer. Ensemble-level crystal structures are cached in a local + * vector. + */ Result<> ComputeCAxisLocations::operator()() { const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); @@ -67,7 +78,8 @@ Result<> ComputeCAxisLocations::operator()() std::vector crystalStructuresBuf(numPhases); crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresBuf.data(), numPhases)); - // Process cells in chunks using bulk I/O + // Process cells in 64K-tuple chunks using sequential bulk I/O. + // This amortizes OOC overhead (page faults, HDF5 reads) over many tuples. constexpr usize k_ChunkSize = 65536; const Eigen::Vector3f cAxis{0.0f, 0.0f, 1.0f}; Eigen::Vector3f c1{0.0f, 0.0f, 0.0f}; @@ -85,7 +97,7 @@ Result<> ComputeCAxisLocations::operator()() const usize chunkCount = std::min(k_ChunkSize, totalPoints - chunkStart); - // Bulk-read quaternions (4 components), phases (1 component) + // Bulk-read this chunk: quaternions (4 components per tuple) and phases (1 per tuple) std::vector quatBuf(chunkCount * 4); std::vector phaseBuf(chunkCount); std::vector outputBuf(chunkCount * 3); @@ -118,6 +130,7 @@ Result<> ComputeCAxisLocations::operator()() } } + // Bulk-write the computed c-axis locations for this chunk outputStore.copyFromBuffer(chunkStart * 3, nonstd::span(outputBuf.data(), chunkCount * 3)); } return result; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.hpp index 18648f84e0..8c9473935e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeCAxisLocations.hpp @@ -9,20 +9,37 @@ namespace nx::core { +/** + * @brief Input values for the ComputeCAxisLocations algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeCAxisLocationsInputValues { - DataPath QuatsArrayPath; - DataPath CellPhasesArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath CAxisLocationsArrayName; + DataPath QuatsArrayPath; ///< Cell-level Float32 quaternions (4 components) + DataPath CellPhasesArrayPath; ///< Cell-level Int32 phase index per voxel + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath CAxisLocationsArrayName; ///< Output: Cell-level Float32 c-axis direction (3 components) }; /** * @class ComputeCAxisLocations - * @brief This filter determines the direction of the C-axis for each Element by applying the quaternion of the Element to the <001> direction, which is the C-axis for Hexagonal materials. This will - * tell where the C-axis of the Element sits in the sample reference frame. + * @brief Converts each voxel's quaternion to a c-axis direction vector in the + * sample reference frame. + * + * For each Element, the quaternion is converted to an orientation matrix, + * transposed (passive to active), and multiplied by the <001> c-axis direction. + * The result is normalized and oriented so the Z component is positive. + * + * Only Hexagonal-High (6/mmm) and Hexagonal-Low (6/m) Laue classes are + * supported; non-hexagonal phases produce NaN output values. + * + * ## OOC Optimization + * + * Cell-level arrays (quaternions, phases) are read in chunks of 65536 tuples + * via `copyIntoBuffer()`, and the output (c-axis locations) is written back + * in matching chunks via `copyFromBuffer()`. Ensemble-level crystal structures + * are cached in a local vector. This replaces per-element `operator[]` access + * that would trigger chunk load/evict cycles with OOC storage. */ - class ORIENTATIONANALYSIS_EXPORT ComputeCAxisLocations { public: @@ -34,6 +51,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeCAxisLocations ComputeCAxisLocations& operator=(const ComputeCAxisLocations&) = delete; ComputeCAxisLocations& operator=(ComputeCAxisLocations&&) noexcept = delete; + /** + * @brief Executes the c-axis location computation using chunked bulk I/O. + * @return Result<> with any errors or warnings encountered. + */ Result<> operator()(); const std::atomic_bool& getCancel(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp index 03f9d869af..393f9c3da9 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.cpp @@ -27,10 +27,22 @@ ComputeFeatureNeighborCAxisMisalignments::ComputeFeatureNeighborCAxisMisalignmen ComputeFeatureNeighborCAxisMisalignments::~ComputeFeatureNeighborCAxisMisalignments() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Computes the c-axis misalignment angle between each pair of neighboring + * hexagonal features. For each feature, the average quaternion is converted to a + * c-axis direction, then the angle between neighboring c-axes is computed and + * stored in a per-feature neighbor list. Optionally computes the average + * misalignment per feature. + * + * OOC strategy: Feature-level arrays (featurePhases, avgQuats) and ensemble-level + * arrays (crystalStructures) are bulk-read into local vectors at startup. The + * main loop operates entirely on these local caches. Output avgCAxisMisalignment + * is accumulated in a local buffer, then bulk-written via copyFromBuffer at the end. + */ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() { // ------------------------------------------------------------------------- - // Cache ensemble-level crystalStructures locally (tiny array) + // Bulk-read ensemble-level crystalStructures into local memory (tiny array). // ------------------------------------------------------------------------- const auto& crystalStructuresStore = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); const usize numPhases = crystalStructuresStore.getNumberOfTuples(); @@ -63,7 +75,9 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() } // ------------------------------------------------------------------------- - // Cache feature-level arrays locally (O(features) — thousands, not millions) + // Bulk-read feature-level arrays into local vectors (O(features) -- thousands, + // not millions of elements). The main loop accesses these randomly by feature + // and neighbor IDs, which would cause OOC thrashing if left in DataStores. // ------------------------------------------------------------------------- const auto& featurePhasesStore = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); const usize totalFeatures = featurePhasesStore.getNumberOfTuples(); @@ -81,7 +95,9 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() auto& cAxisMisalignmentList = m_DataStructure.getDataRefAs>(m_InputValues->CAxisMisalignmentListArrayName); // ------------------------------------------------------------------------- - // Output buffer for avgCAxisMisalignment — accumulate locally, bulk-write at end + // Local accumulation buffer for avgCAxisMisalignment. Using a local vector + // avoids per-element OOC writes during the neighbor loop. The final result + // is bulk-written to the DataStore at the end. // ------------------------------------------------------------------------- Float32Array* avgCAxisMisalignmentPtr = nullptr; std::vector avgCAxisBuf; @@ -190,7 +206,8 @@ Result<> ComputeFeatureNeighborCAxisMisalignments::operator()() } // ------------------------------------------------------------------------- - // Bulk-write the avgCAxisMisalignment output buffer back to the DataStore + // Single bulk-write of the completed avgCAxisMisalignment buffer back to the + // DataStore. All accumulation and normalization was done in local RAM. // ------------------------------------------------------------------------- if(m_InputValues->FindAvgMisals) { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.hpp index 0a9bd14316..6656bc643a 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureNeighborCAxisMisalignments.hpp @@ -9,22 +9,40 @@ namespace nx::core { +/** + * @brief Input values for the ComputeFeatureNeighborCAxisMisalignments algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeFeatureNeighborCAxisMisalignmentsInputValues { - bool FindAvgMisals; - DataPath NeighborListArrayPath; - DataPath AvgQuatsArrayPath; - DataPath FeaturePhasesArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath CAxisMisalignmentListArrayName; - DataPath AvgCAxisMisalignmentsArrayName; + bool FindAvgMisals; ///< If true, also compute the average misalignment per feature + DataPath NeighborListArrayPath; ///< Feature-level NeighborList of neighbor feature IDs + DataPath AvgQuatsArrayPath; ///< Feature-level Float32 average quaternions (4 components) + DataPath FeaturePhasesArrayPath; ///< Feature-level Int32 phase index per feature + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath CAxisMisalignmentListArrayName; ///< Output: Feature-level NeighborList of c-axis misalignment angles (degrees) + DataPath AvgCAxisMisalignmentsArrayName; ///< Output: Feature-level Float32 average c-axis misalignment (degrees) }; /** * @class ComputeFeatureNeighborCAxisMisalignments - * @brief This filter determines, for each Feature, the C-axis mis alignments with the Features that are in contact with it. + * @brief Computes the c-axis misalignment angle between each Feature and its + * neighbors, plus optionally the per-Feature average misalignment. + * + * For each pair of neighboring features that share the same Hexagonal-High + * phase, the c-axis direction of each feature is computed from its average + * quaternion. The angle between the two c-axis directions gives the + * misalignment, stored in degrees. + * + * ## OOC Optimization + * + * Feature-level arrays (phases, avgQuats) and ensemble-level crystal structures + * are cached entirely in local vectors via `copyIntoBuffer()` at algorithm + * start. The average misalignment output is accumulated in a local buffer and + * written back via `copyFromBuffer()` at the end. Since this algorithm operates + * on feature-level data (not cell-level), the arrays are typically small enough + * to cache entirely, but using bulk I/O still avoids per-element virtual + * dispatch overhead in the hot loop. */ - class ORIENTATIONANALYSIS_EXPORT ComputeFeatureNeighborCAxisMisalignments { public: @@ -37,6 +55,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeFeatureNeighborCAxisMisalignments ComputeFeatureNeighborCAxisMisalignments& operator=(const ComputeFeatureNeighborCAxisMisalignments&) = delete; ComputeFeatureNeighborCAxisMisalignments& operator=(ComputeFeatureNeighborCAxisMisalignments&&) noexcept = delete; + /** + * @brief Executes the c-axis misalignment computation with locally cached data. + * @return Result<> with any errors or warnings (e.g., non-hexagonal phases). + */ Result<> operator()(); private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp index 518aff28a1..0cad95a32d 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.cpp @@ -34,12 +34,24 @@ ComputeFeatureReferenceCAxisMisorientations::ComputeFeatureReferenceCAxisMisorie ComputeFeatureReferenceCAxisMisorientations::~ComputeFeatureReferenceCAxisMisorientations() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Computes each cell's c-axis misorientation relative to its feature's + * average c-axis (hexagonal phases only). Also computes per-feature mean and + * standard deviation of these misorientations. + * + * OOC strategy: Uses Z-slice-based bulk I/O. For each Z-plane, all cell-level + * input arrays are read in one copyIntoBuffer call per array, processed, and + * the per-cell output is written back with copyFromBuffer. Feature-level arrays + * (avgCAxes, crystalStructures) are cached entirely in local vectors since + * they are accessed randomly by featureId/phase index. A second Z-slice pass + * re-reads the output to compute the standard deviation. + */ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() { /* ************************************************************************** - * Cache ensemble-level crystalStructures locally (tiny array, avoids - * per-element OOC overhead during the main cell loop) + * Bulk-read ensemble-level crystalStructures into local memory (tiny array). + * Avoids per-element OOC virtual dispatch during the main cell loop. */ const auto& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath); const usize numCrystalStructures = crystalStructures.getNumberOfTuples(); @@ -71,21 +83,22 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() } /* ************************************************************************** - * Get DataStore references for bulk I/O + * Obtain DataStore references for cell-level bulk I/O. All cell reads go + * through copyIntoBuffer (one Z-slice at a time) rather than operator[]. */ - // Input Cell Data (DataStore refs for copyIntoBuffer) const auto& featureIdsStore = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath).getDataStoreRef(); const auto& quatsStore = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath).getDataStoreRef(); const auto& cellPhasesStore = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath).getDataStoreRef(); - // Input Feature Data — cache avgCAxes locally (feature-level, small) + // Cache avgCAxes locally — accessed randomly by featureId in the cell loop. + // Feature count is O(thousands) so this fits comfortably in RAM. const auto& avgCAxes = m_DataStructure.getDataRefAs(m_InputValues->AvgCAxesArrayPath); const usize totalFeatures = avgCAxes.getNumberOfTuples(); const usize avgCAxesSize = totalFeatures * 3; std::vector avgCAxesLocal(avgCAxesSize); avgCAxes.getDataStoreRef().copyIntoBuffer(0, nonstd::span(avgCAxesLocal.data(), avgCAxesSize)); - // Output Cell Data (DataStore ref for copyFromBuffer) + // Output cell DataStore — written one Z-slice at a time via copyFromBuffer auto& cellRefCAxisMisStore = m_DataStructure.getDataRefAs(m_InputValues->FeatureReferenceCAxisMisorientationsArrayPath).getDataStoreRef(); // Output Feature Data @@ -109,7 +122,8 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() const Eigen::Vector3d cAxis{0.0, 0.0, 1.0}; - // Z-slice buffers for cell-level arrays (avoids per-element OOC access) + // Z-slice buffers: one slice of each cell-level array is read/written per + // Z-plane iteration. This converts random 3D access into sequential slice I/O. std::vector featureIdSlice(sliceSize); std::vector cellPhaseSlice(sliceSize); std::vector quatSlice(quatSliceSize); @@ -206,8 +220,9 @@ Result<> ComputeFeatureReferenceCAxisMisorientations::operator()() featAvgCAxisMis[featureId] = avgMisorientations[featureId] / static_cast(counts[featureId]); } - // These 2 loops compute the population standard deviation of those misorientations for - // each feature. Re-read cell data one Z-slice at a time. + // Compute the population standard deviation of misorientations per feature. + // This requires a second pass over cell data. We re-read featureIds and the + // just-written output array one Z-slice at a time (sequential OOC access). std::vector stdevs(totalFeatures, 0.0); for(int64 plane = 0; plane < zPoints; plane++) { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.hpp index e3d287f7d0..9bb8383fb2 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceCAxisMisorientations.hpp @@ -11,34 +11,57 @@ namespace nx::core { +/** + * @brief Input values for the ComputeFeatureReferenceCAxisMisorientations algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeFeatureReferenceCAxisMisorientationsInputValues { // Input Geometry - DataPath ImageGeometryPath; + DataPath ImageGeometryPath; ///< ImageGeom providing the voxel grid dimensions + // Input Cell Data - DataPath FeatureIdsArrayPath; - DataPath CellPhasesArrayPath; - DataPath QuatsArrayPath; + DataPath FeatureIdsArrayPath; ///< Cell-level Int32 feature ID per voxel + DataPath CellPhasesArrayPath; ///< Cell-level Int32 phase index per voxel + DataPath QuatsArrayPath; ///< Cell-level Float32 quaternions (4 components) // Input Feature Data - DataPath AvgCAxesArrayPath; + DataPath AvgCAxesArrayPath; ///< Feature-level Float32 average c-axis (3 components) // Input Ensemble Data - DataPath CrystalStructuresArrayPath; + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes // Output Cell Data - DataPath FeatureReferenceCAxisMisorientationsArrayPath; + DataPath FeatureReferenceCAxisMisorientationsArrayPath; ///< Output: Cell-level Float32 c-axis misorientation (degrees) // Output Feature Data - DataPath FeatureAvgCAxisMisorientationsArrayPath; - DataPath FeatureStdevCAxisMisorientationsArrayPath; + DataPath FeatureAvgCAxisMisorientationsArrayPath; ///< Output: Feature-level Float32 average c-axis misorientation + DataPath FeatureStdevCAxisMisorientationsArrayPath; ///< Output: Feature-level Float32 standard deviation }; /** * @class ComputeFeatureReferenceCAxisMisorientations - * @brief This filter calculates the misorientation angle between the C-axis of each Cell within a Feature and the average C-axis for that Feature and stores that value for each Cell. + * @brief Computes the misorientation angle between each voxel's c-axis and + * the average c-axis of its Feature, plus per-Feature mean and standard + * deviation of those angles. + * + * Only Hexagonal-High (6/mmm) and Hexagonal-Low (6/m) Laue classes are + * supported; non-hexagonal phases are skipped with zero output. + * + * ## OOC Optimization + * + * The algorithm processes one Z-slice at a time. For each slice: + * 1. Cell-level arrays (featureIds, phases, quats) are bulk-read into + * local buffers via `copyIntoBuffer()`. + * 2. Feature-level avgCAxes and ensemble-level crystal structures are + * cached in local vectors at algorithm start (small arrays). + * 3. The per-cell output is accumulated in a local buffer and + * bulk-written via `copyFromBuffer()`. + * 4. A second Z-slice pass re-reads the output to compute the + * per-Feature standard deviation. + * + * This Z-slice strategy gives predictable memory usage (one slice at a time) + * and sequential I/O patterns that perform well with OOC chunked storage. */ - class ORIENTATIONANALYSIS_EXPORT ComputeFeatureReferenceCAxisMisorientations { public: @@ -51,6 +74,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeFeatureReferenceCAxisMisorientations ComputeFeatureReferenceCAxisMisorientations& operator=(const ComputeFeatureReferenceCAxisMisorientations&) = delete; ComputeFeatureReferenceCAxisMisorientations& operator=(ComputeFeatureReferenceCAxisMisorientations&&) noexcept = delete; + /** + * @brief Executes the c-axis misorientation computation using Z-slice bulk I/O. + * @return Result<> with any errors or warnings encountered. + */ Result<> operator()(); private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp index 3e21161b28..632ff2683e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.cpp @@ -40,6 +40,18 @@ const std::atomic_bool& ComputeFeatureReferenceMisorientations::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Computes the misorientation between each cell's quaternion and a + * reference orientation for its feature. Two reference modes are supported: + * Mode 0: use the feature's average quaternion (from a prior filter). + * Mode 1: use the quaternion of the voxel farthest from the grain boundary + * (the "center" voxel, found via grain boundary Euclidean distances). + * + * OOC strategy: All cell-level arrays are read in 64K-tuple chunks via + * copyIntoBuffer. Feature-level and ensemble-level arrays are cached entirely + * in local vectors at startup (small enough to fit in RAM). Misorientation + * output is accumulated in a chunk buffer and bulk-written via copyFromBuffer. + */ Result<> ComputeFeatureReferenceMisorientations::operator()() { DataPath imageGeomPath = m_InputValues->CellPhasesArrayPath.getParent().getParent(); @@ -80,12 +92,14 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() return MakeErrorResult(-34900, "Total features was zero. The filter cannot proceed. Check either the feature attribute matrix or the average quaternions for proper size"); } - // Cache crystal structures locally (ensemble-level, tiny) + // Bulk-read ensemble-level crystal structures (typically < 10 entries) into + // local memory to avoid per-element OOC virtual dispatch in the cell loop. const usize numXtalEntries = crystalStructures.getNumberOfTuples(); std::vector localCrystalStructures(numXtalEntries); crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(localCrystalStructures.data(), numXtalEntries)); - // Cache avgQuats locally (feature-level) — used in mode 0 + // Cache average quaternions locally when using mode 0 (feature average). + // This avoids random-access OOC reads during the main cell loop. std::vector localAvgQuats; if(m_InputValues->ReferenceOrientation == 0 && avgQuatsPtr != nullptr) { @@ -102,7 +116,9 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() const auto& quatsStore = quats.getDataStoreRef(); auto& misoStore = featureReferenceMisorientations.getDataStoreRef(); - // Mode 1: find center voxels using chunked I/O + // Mode 1: find the center voxel for each feature — the voxel with the largest + // grain boundary Euclidean distance. Uses chunked sequential reads of both + // featureIds and GB distances to avoid random OOC access. if(m_InputValues->ReferenceOrientation == 1) { const auto& gbDistStore = m_DataStructure.getDataRefAs(m_InputValues->GBEuclideanDistancesArrayPath).getDataStoreRef(); @@ -136,6 +152,9 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() euclideanCellCenters->setTuple(i, cellCenter.data()); } + // Cache the quaternion at each feature's center voxel. These are point + // reads from the quats store (one per feature), so we read them individually + // rather than reading the entire quats array into RAM. centerQuats.resize(totalFeatures * 4, 0.0f); for(usize i = 1; i < totalFeatures; i++) { @@ -148,15 +167,19 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() } } + // Accumulators for computing per-feature average misorientation std::vector avgMisorientationSums(totalFeatures, 0.0f); std::vector avgMisorientationCounts(totalFeatures, 0.0f); featureReferenceMisorientations.fill(0.0f); + // Pre-allocate chunk I/O buffers for the main misorientation computation loop. + // The misoBuf accumulates output values per chunk, then is bulk-written. auto featureIdBuf = std::make_unique(k_ChunkTuples); auto phasesBuf = std::make_unique(k_ChunkTuples); auto quatsBuf = std::make_unique(k_ChunkTuples * 4); auto misoBuf = std::make_unique(k_ChunkTuples); + // Main cell loop — sequential chunked reads of cell data, chunked writes of output for(usize offset = 0; offset < totalVoxels; offset += k_ChunkTuples) { if(m_ShouldCancel) @@ -197,9 +220,11 @@ Result<> ComputeFeatureReferenceMisorientations::operator()() avgMisorientationSums[featureId] += misoValue; } } + // Bulk-write this chunk's misorientation values to the output DataStore misoStore.copyFromBuffer(offset, nonstd::span(misoBuf.get(), count)); } + // Compute per-feature average misorientation from the accumulated sums avgReferenceMisorientation[0] = 0.0f; for(usize featureIdx = 1; featureIdx < totalFeatures; featureIdx++) { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.hpp index 9cd36f1be1..d6d1ccc567 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeFeatureReferenceMisorientations.hpp @@ -10,23 +10,43 @@ namespace nx::core { +/** + * @brief Input values for the ComputeFeatureReferenceMisorientations algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeFeatureReferenceMisorientationsInputValues { - ChoicesParameter::ValueType ReferenceOrientation; - DataPath FeatureAttributeMatrixPath; - DataPath FeatureIdsArrayPath; - DataPath CellPhasesArrayPath; - DataPath QuatsArrayPath; - DataPath GBEuclideanDistancesArrayPath; - DataPath AvgQuatsArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath FeatureReferenceMisorientationsArrayName; - DataPath FeatureAvgMisorientationsArrayName; - DataPath FeatureEuclideanCentersPath; + ChoicesParameter::ValueType ReferenceOrientation; ///< 0 = average orientation, 1 = orientation farthest from boundary + DataPath FeatureAttributeMatrixPath; ///< Feature-level AttributeMatrix (used for tuple count in mode 1) + DataPath FeatureIdsArrayPath; ///< Cell-level Int32 feature ID per voxel + DataPath CellPhasesArrayPath; ///< Cell-level Int32 phase index per voxel + DataPath QuatsArrayPath; ///< Cell-level Float32 quaternions (4 components) + DataPath GBEuclideanDistancesArrayPath; ///< Cell-level Float32 grain boundary Euclidean distances (mode 1 only) + DataPath AvgQuatsArrayPath; ///< Feature-level Float32 average quaternions (mode 0 only) + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath FeatureReferenceMisorientationsArrayName; ///< Output: Cell-level Float32 misorientation angle (degrees) + DataPath FeatureAvgMisorientationsArrayName; ///< Output: Feature-level Float32 average misorientation (degrees) + DataPath FeatureEuclideanCentersPath; ///< Output: Feature-level Float32 Euclidean center coordinates (mode 1) }; /** - * @class + * @class ComputeFeatureReferenceMisorientations + * @brief Computes the misorientation angle between each voxel and its Feature's + * reference orientation, plus the per-Feature average of those angles. + * + * Two reference modes are supported: + * - **Mode 0**: Reference is the Feature's average quaternion (from AvgQuats). + * - **Mode 1**: Reference is the voxel farthest from the grain boundary + * (identified by maximum grain-boundary Euclidean distance). + * + * ## OOC Optimization + * + * All cell-level arrays (featureIds, phases, quats, GB distances) are read + * in chunks of 65536 tuples via `copyIntoBuffer()`. Feature-level arrays + * (avgQuats, crystal structures) are cached entirely in local vectors at + * algorithm start. The cell-level misorientation output is written back in + * matching chunks via `copyFromBuffer()`. In mode 1, the center-voxel + * identification pass also uses chunked I/O. This strategy converts per-element + * virtual dispatch into bulk sequential I/O, eliminating OOC performance cliffs. */ class ORIENTATIONANALYSIS_EXPORT ComputeFeatureReferenceMisorientations { @@ -40,6 +60,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeFeatureReferenceMisorientations ComputeFeatureReferenceMisorientations& operator=(const ComputeFeatureReferenceMisorientations&) = delete; ComputeFeatureReferenceMisorientations& operator=(ComputeFeatureReferenceMisorientations&&) noexcept = delete; + /** + * @brief Executes the misorientation computation using chunked bulk I/O. + * @return Result<> with any errors encountered during execution. + */ Result<> operator()(); const std::atomic_bool& getCancel(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp index 83848bbc70..c2605bce77 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.cpp @@ -22,8 +22,12 @@ namespace const usize k_NumMisoReps = 576 * 4; } /** - * @brief The CalculateGBCDImpl class implements a threaded algorithm that calculates the - * grain boundary character distribution (GBCD) for a surface mesh + * @brief Parallel worker that computes GBCD bin indices for a chunk of surface + * mesh triangles. Accepts raw pointers to locally cached feature-level (Euler + * angles, phases) and ensemble-level (crystal structures) data, plus + * offset-adjusted pointers to the current chunk of triangle labels and normals. + * All data access is through raw pointers into local buffers -- zero OOC + * virtual dispatch in the parallel hot loop. */ class CalculateGBCDImpl { @@ -363,6 +367,23 @@ const std::atomic_bool& ComputeGBCD::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Computes the Grain Boundary Character Distribution (GBCD) by + * iterating over all triangle faces on the surface mesh in chunks of 50K + * triangles. For each chunk, the CalculateGBCDImpl parallel worker computes + * GBCD bin indices for each triangle, then the main loop accumulates + * face-area-weighted contributions into the GBCD histogram. Finally, the + * histogram is normalized to multiples of random distribution (MRD). + * + * OOC strategy: Feature-level arrays (Euler angles, phases) and ensemble-level + * arrays (crystal structures) are bulk-read into local vectors at startup since + * the parallel worker accesses them randomly by feature ID. Triangle-level + * arrays (labels, normals, areas) are chunk-read per iteration via + * copyIntoBuffer. The GBCD output histogram is accumulated in a local buffer + * and bulk-written to the DataStore at the end via copyFromBuffer. + * The CalculateGBCDImpl worker receives raw pointers into the local caches, + * eliminating all OOC virtual dispatch from the parallel hot loop. + */ Result<> ComputeGBCD::operator()() { auto& eulerAngles = m_DataStructure.getDataRefAs(m_InputValues->FeatureEulerAnglesArrayPath); @@ -374,7 +395,9 @@ Result<> ComputeGBCD::operator()() auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayName); - // Cache feature-level arrays locally to eliminate per-element OOC overhead + // Bulk-read feature-level arrays into local vectors. The parallel worker + // accesses these randomly by feature ID (from triangle face labels), which + // would cause severe OOC chunk thrashing if left in DataStores. const usize numEulerElements = eulerAngles.getSize(); std::vector eulersCache(numEulerElements); eulerAngles.getDataStoreRef().copyIntoBuffer(0, nonstd::span(eulersCache.data(), numEulerElements)); @@ -383,7 +406,7 @@ Result<> ComputeGBCD::operator()() std::vector phasesCache(numPhaseElements); phases.getDataStoreRef().copyIntoBuffer(0, nonstd::span(phasesCache.data(), numPhaseElements)); - // Cache ensemble-level arrays (tiny) + // Bulk-read ensemble-level crystal structures (tiny, typically < 10 entries) usize totalPhases = crystalStructures.getNumberOfTuples(); std::vector crystalStructuresCache(totalPhases); crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.data(), totalPhases)); @@ -406,7 +429,7 @@ Result<> ComputeGBCD::operator()() messageHelper.sendMessage("1/2 Starting GBCD Calculation and Summation Phase"); ThrottledMessenger throttledMessenger = messageHelper.createThrottledMessenger(); - // Pre-allocate chunk buffers for triangle-level arrays + // Pre-allocate chunk buffers for triangle-level arrays (reused each iteration) const auto& labelsStore = faceLabels.getDataStoreRef(); const auto& normalsStore = faceNormals.getDataStoreRef(); const auto& areasStore = faceAreas.getDataStoreRef(); @@ -414,8 +437,9 @@ Result<> ComputeGBCD::operator()() std::vector normalsBuf(triangleChunkSize * 3); std::vector areasBuf(triangleChunkSize); - // Cache the full GBCD output locally for accumulation — bounded by - // totalPhases * totalGBCDBins (bin resolution, not cell count) + // Local GBCD histogram accumulator. Size is bounded by totalPhases * + // totalGBCDBins (determined by angular resolution, not by cell count), + // so it fits in RAM even for multi-phase datasets. const usize gbcdTotalElements = gbcd.getSize(); std::vector gbcdBuf(gbcdTotalElements, 0.0); @@ -433,7 +457,9 @@ Result<> ComputeGBCD::operator()() sizeGbcd.initializeBinsWithValue(-1); sizeGbcd.m_GbcdHemiCheck.assign(sizeGbcd.m_GbcdHemiCheck.size(), false); - // Chunk-read triangle arrays for this iteration + // Bulk-read this chunk of triangle data (labels, normals, areas). + // The parallel worker receives offset-adjusted raw pointers into these + // buffers so it can index using absolute triangle indices. labelsStore.copyIntoBuffer(i * 2, nonstd::span(labelsBuf.data(), triangleChunkSize * 2)); normalsStore.copyIntoBuffer(i * 3, nonstd::span(normalsBuf.data(), triangleChunkSize * 3)); areasStore.copyIntoBuffer(i, nonstd::span(areasBuf.data(), triangleChunkSize)); @@ -486,7 +512,8 @@ Result<> ComputeGBCD::operator()() messageHelper.sendMessage("2/2 Starting GBCD Normalization Phase"); - // Normalize GBCD in the local buffer, then write back to the DataStore + // Normalize the GBCD histogram to MRD (multiples of random distribution) + // in the local buffer, then bulk-write the final result to the DataStore. for(usize i = 0; i < totalPhases; i++) { const usize k_PhaseShift = i * static_cast(totalGBCDBins); @@ -496,6 +523,7 @@ Result<> ComputeGBCD::operator()() gbcdBuf[k_PhaseShift + j] *= k_MrdFactor; } } + // Single bulk-write of the normalized GBCD histogram to the output DataStore gbcd.getDataStoreRef().copyFromBuffer(0, nonstd::span(gbcdBuf.data(), gbcdTotalElements)); return {}; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.hpp index 773dc96e02..85e061b247 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCD.hpp @@ -25,22 +25,47 @@ struct SizeGBCD usize m_NumMisoReps; }; +/** + * @brief Input values for the ComputeGBCD algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeGBCDInputValues { - float32 GBCDRes; - DataPath TriangleGeometryPath; - DataPath SurfaceMeshFaceLabelsArrayPath; - DataPath SurfaceMeshFaceNormalsArrayPath; - DataPath SurfaceMeshFaceAreasArrayPath; - DataPath FeatureEulerAnglesArrayPath; - DataPath FeaturePhasesArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath FaceEnsembleAttributeMatrixName; - DataPath GBCDArrayName; + float32 GBCDRes; ///< GBCD resolution in degrees + DataPath TriangleGeometryPath; ///< TriangleGeom containing the surface mesh + DataPath SurfaceMeshFaceLabelsArrayPath; ///< Face-level Int32 labels (2 components: feature1, feature2) + DataPath SurfaceMeshFaceNormalsArrayPath; ///< Face-level Float64 normals (3 components) + DataPath SurfaceMeshFaceAreasArrayPath; ///< Face-level Float64 areas + DataPath FeatureEulerAnglesArrayPath; ///< Feature-level Float32 Euler angles (3 components) + DataPath FeaturePhasesArrayPath; ///< Feature-level Int32 phase index per feature + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath FaceEnsembleAttributeMatrixName; ///< Ensemble-level AttributeMatrix for output + DataPath GBCDArrayName; ///< Output: Ensemble-level Float64 GBCD histogram }; /** - * @class + * @class ComputeGBCD + * @brief Computes the five-dimensional Grain Boundary Character Distribution + * (GBCD) for a triangle surface mesh. + * + * The GBCD represents the relative area of grain boundary for a given + * misorientation and boundary normal. Triangles are processed in chunks, + * and for each triangle the misorientation bin and boundary normal bin are + * determined. The triangle's area is accumulated into the appropriate GBCD + * bin. The final distribution is normalized to multiples of random distribution + * (MRD). + * + * ## OOC Optimization + * + * Several levels of caching eliminate per-element OOC overhead: + * - Feature-level arrays (Euler angles, phases) are cached entirely in + * local vectors via `copyIntoBuffer()` -- these are small (O(features)). + * - Ensemble-level crystal structures are cached locally (tiny). + * - Triangle-level arrays (labels, normals, areas) are read in chunks of + * 50000 triangles via `copyIntoBuffer()` before each parallel pass. + * - The GBCD output histogram is accumulated in a local vector and written + * back via `copyFromBuffer()` after normalization. + * - The parallel `CalculateGBCDImpl` worker receives raw pointers into + * the local buffers, achieving zero virtual dispatch in the hot loop. */ class ORIENTATIONANALYSIS_EXPORT ComputeGBCD { @@ -53,6 +78,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeGBCD ComputeGBCD& operator=(const ComputeGBCD&) = delete; ComputeGBCD& operator=(ComputeGBCD&&) noexcept = delete; + /** + * @brief Executes the GBCD computation with locally cached data and chunked I/O. + * @return Result<> with any errors encountered during execution. + */ Result<> operator()(); const std::atomic_bool& getCancel(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp index b3bfe85728..daf2640c3c 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDMetricBased.cpp @@ -58,8 +58,13 @@ struct TriAreaAndNormals }; /** - * @brief The TrianglesSelector class implements a threaded algorithm that determines which triangles to - * include in the GBCD calculation + * @brief Parallel worker that selects triangles matching the specified + * misorientation for the metric-based GBCD calculation. Receives raw pointers + * to locally cached feature-level Euler angles and phases (pre-read via + * copyIntoBuffer), plus the resolved crystal structure for the phase of + * interest. This eliminates OOC virtual dispatch during the parallel loop. + * Triangle-level arrays (faceLabels, faceNormals, faceAreas) are still + * accessed through DataArray references because they are read sequentially. */ class TrianglesSelector { @@ -342,6 +347,19 @@ const std::atomic_bool& ComputeGBCDMetricBased::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Computes the Grain Boundary Character Distribution using a metric-based + * approach. Triangles matching a specified misorientation (within tolerance) are + * selected, then the distribution is evaluated at sampling points on the unit + * hemisphere using a kernel density estimator. + * + * OOC strategy: Feature-level arrays (Euler angles, phases) and ensemble-level + * arrays (crystal structures) are bulk-read into local vectors at startup. The + * TrianglesSelector parallel worker receives raw pointers to these caches, + * eliminating OOC virtual dispatch. Triangle-level arrays (face labels, areas) + * are chunk-read per iteration for the totalFaceArea accumulation. Feature-face + * labels are also cached locally for the distinct boundary count loop. + */ Result<> ComputeGBCDMetricBased::operator()() { // -------------------- check if directories are ok and if output files can be opened ----------- @@ -395,7 +413,9 @@ Result<> ComputeGBCDMetricBased::operator()() auto& triangleGeom = m_DataStructure.getDataRefAs(m_InputValues->TriangleGeometryPath); const IGeometry::SharedFaceList& triangles = triangleGeom.getFacesRef(); - // Cache feature-level arrays locally to eliminate per-element OOC overhead + // Bulk-read feature-level arrays (Euler angles, phases) into local vectors. + // The parallel TrianglesSelector accesses these randomly by feature ID from + // triangle face labels — leaving them in OOC DataStores would cause thrashing. const usize numEulerElements = eulerAngles.getSize(); std::vector eulerCache(numEulerElements); eulerAngles.getDataStoreRef().copyIntoBuffer(0, nonstd::span(eulerCache.data(), numEulerElements)); @@ -404,12 +424,13 @@ Result<> ComputeGBCDMetricBased::operator()() std::vector phasesCache(numPhaseElements); phases.getDataStoreRef().copyIntoBuffer(0, nonstd::span(phasesCache.data(), numPhaseElements)); - // Cache ensemble-level arrays (tiny) + // Bulk-read ensemble-level crystal structures (tiny, typically < 10 entries) const usize numCrystalStructures = crystalStructures.getSize(); std::vector crystalStructuresCache(numCrystalStructures); crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.data(), numCrystalStructures)); - // Cache feature-face labels (feature-level) + // Bulk-read feature-face labels for the distinct boundary count loop below. + // This is O(feature_faces) which is much smaller than O(mesh_triangles). const usize numFeatureFaceElements = featureFaceLabels.getSize(); std::vector featureFaceLabelsCache(numFeatureFaceElements); featureFaceLabels.getDataStoreRef().copyIntoBuffer(0, nonstd::span(featureFaceLabelsCache.data(), numFeatureFaceElements)); @@ -491,8 +512,10 @@ Result<> ComputeGBCDMetricBased::operator()() triChunkSize = numMeshTriangles; } - // Accumulate totalFaceArea per-chunk by re-checking the geometric filter - // conditions instead of storing an O(n) triIncluded mask. + // Accumulate totalFaceArea by re-checking geometric filter conditions per + // chunk, rather than storing an O(n) triIncluded mask. This trades a small + // amount of redundant computation for eliminating a large boolean array + // that would also need OOC-safe access patterns. float64 totalFaceArea = 0.0; for(usize i = 0; i < numMeshTriangles; i += triChunkSize) @@ -515,7 +538,8 @@ Result<> ComputeGBCDMetricBased::operator()() dataAlg.execute(GBCDMetricBased::TrianglesSelector(m_InputValues->ExcludeTripleLines, triangles, nodeTypes, selectedTriangles, misResolution, m_InputValues->PhaseOfInterest, gFixedT, crystalStructuresCache[m_InputValues->PhaseOfInterest], eulerCache.data(), phasesCache.data(), faceLabels, faceNormals, faceAreas)); - // Chunk-read triangle arrays for totalFaceArea accumulation + // Bulk-read this chunk's face labels and areas for totalFaceArea accumulation. + // Re-applies the same geometric filter conditions that TrianglesSelector uses. { std::vector labelsBuf(currentChunkSize * 2); std::vector areasBuf(currentChunkSize); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp index 00622341a9..f4fca9c0d1 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.cpp @@ -15,18 +15,41 @@ using namespace nx::core; namespace { +/** + * @class ComputeGBCDPoleFigureImpl + * @brief Threaded worker for generating a GBCD stereographic pole figure. + * + * Each instance is invoked by ParallelData2DAlgorithm on a disjoint 2D rectangular + * region of the output pole figure image. For each pixel in the assigned region, the + * worker: + * + * 1. Performs inverse stereographic projection to obtain a unit-sphere direction. + * 2. Iterates over all pairs of crystal symmetry operators (O(nSym^2) per pixel). + * 3. For each symmetry pair, computes the symmetrically-equivalent misorientation + * in both crystal reference frames. + * 4. If the equivalent misorientation falls within the fundamental zone (all three + * Euler angles < pi/2), the corresponding 5D GBCD bin is looked up and accumulated. + * 5. The pixel intensity is the average GBCD value across all valid symmetry pairs. + * + * The GBCD is stored as a 5D histogram with dimensions: + * [misorientation_phi1, cos(misorientation_Phi), misorientation_phi2, boundary_normal_theta, boundary_normal_phi] + * multiplied by 2 for the two hemispheres (northern and southern). + * + * This worker operates on raw float64 pointers to locally-cached data, making it + * safe for multi-threaded execution. No OOC DataStore access occurs in the hot loop. + */ class ComputeGBCDPoleFigureImpl { private: - float64* m_PoleFigure; - std::array m_Dimensions; - ebsdlib::LaueOps::Pointer m_OrientOps; - const std::vector& m_GbcdDeltas; - const std::vector& m_GbcdLimits; - const std::vector& m_GbcdSizes; - const float64* m_Gbcd; - int32 m_PhaseOfInterest = 0; - const std::vector& m_MisorientationRotation; + float64* m_PoleFigure; ///< Output pole figure pixel intensities (xPoints * yPoints). + std::array m_Dimensions; ///< [xPoints, yPoints] of the output image. + ebsdlib::LaueOps::Pointer m_OrientOps; ///< LaueOps for the crystal structure of the phase of interest. + const std::vector& m_GbcdDeltas; ///< Bin width in each of the 5 GBCD dimensions. + const std::vector& m_GbcdLimits; ///< Lower [0-4] and upper [5-9] bounds for the 5 GBCD dimensions. + const std::vector& m_GbcdSizes; ///< Number of bins in each of the 5 GBCD dimensions. + const float64* m_Gbcd; ///< Pointer to the (possibly phase-offset) GBCD data. + int32 m_PhaseOfInterest = 0; ///< Phase index offset applied when indexing into m_Gbcd. + const std::vector& m_MisorientationRotation; ///< User-specified misorientation [angle_deg, axis_x, axis_y, axis_z]. public: ComputeGBCDPoleFigureImpl(float64* poleFigurePtr, const std::array& dimensions, const ebsdlib::LaueOps::Pointer& orientOps, const std::vector& gbcdDeltasArray, @@ -45,6 +68,12 @@ class ComputeGBCDPoleFigureImpl } ~ComputeGBCDPoleFigureImpl() = default; + /** + * @brief Generates pole figure intensities for the pixel sub-region [xStart,xEnd) x [yStart,yEnd). + * + * For each pixel within the unit circle of the stereographic projection, computes the + * average GBCD intensity across all symmetrically-equivalent misorientations. + */ void generate(usize xStart, usize xEnd, usize yStart, usize yEnd) const { ebsdlib::Matrix3X1 vec = {0.0f, 0.0f, 0.0f}; @@ -61,17 +90,21 @@ class ComputeGBCDPoleFigureImpl ebsdlib::Matrix3X3 sym2t; // = {{0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}, {0.0f, 0.0f, 0.0f}}; // Matrix3X1 misEuler1 = {0.0f, 0.0f, 0.0f}; + // Convert the user-specified misorientation from axis-angle (degrees) to a + // 3x3 rotation matrix (dg). The transpose (dgt) represents the inverse + // misorientation, used when repeating the lookup in the second crystal frame. float32 misAngle = m_MisorientationRotation[0] * nx::core::Constants::k_PiOver180F; nx::core::FloatVec3 normAxis = {m_MisorientationRotation[1], m_MisorientationRotation[2], m_MisorientationRotation[3]}; normAxis = normAxis.normalize(); - // convert axis angle to matrix representation of misorientation ebsdlib::Matrix3X3 dg = ebsdlib::AxisAngleFType(normAxis[0], normAxis[1], normAxis[2], misAngle).toOrientationMatrix().toGMatrix(); - // take inverse of misorientation variable to use for switching symmetry ebsdlib::Matrix3X3 dgt = dg.transpose(); - // get number of symmetry operators + // Number of crystal symmetry operators for this Laue class (e.g., 24 for cubic). int32 nSym = m_OrientOps->getNumSymOps(); + // Output image grid parameters. The pole figure spans [-1, 1] in both x and y, + // centered at (xPointsHalf, yPointsHalf). Only pixels inside the unit circle + // (x^2 + y^2 <= 1) are computed. int32 xPoints = m_Dimensions[0]; int32 yPoints = m_Dimensions[1]; int32 xPointsHalf = xPoints / 2; @@ -81,11 +114,16 @@ class ComputeGBCDPoleFigureImpl bool nhCheck = false; int32 hemisphere = 0; + // Precompute stride multipliers for the 5D GBCD array's row-major linearization. + // The 5D index [loc1, loc2, loc3, loc4, loc5] maps to: + // loc1 + loc2*shift1 + loc3*shift2 + loc4*shift3 + loc5*shift4 + // with a final *2 + hemisphere for the north/south hemisphere selection. int32 shift1 = m_GbcdSizes[0]; int32 shift2 = m_GbcdSizes[0] * m_GbcdSizes[1]; int32 shift3 = m_GbcdSizes[0] * m_GbcdSizes[1] * m_GbcdSizes[2]; int32 shift4 = m_GbcdSizes[0] * m_GbcdSizes[1] * m_GbcdSizes[2] * m_GbcdSizes[3]; + // Total number of GBCD bins per phase (both hemispheres). int64 totalGbcdBins = m_GbcdSizes[0] * m_GbcdSizes[1] * m_GbcdSizes[2] * m_GbcdSizes[3] * m_GbcdSizes[4] * 2; std::vector dims = {1ULL}; @@ -102,22 +140,31 @@ class ComputeGBCDPoleFigureImpl { double sum = 0.0; int32 count = 0; + // Inverse stereographic projection: map (x, y) in the unit disk to a + // unit-sphere direction (vec). This is the boundary-plane normal direction + // in the sample reference frame. vec[2] = -((x * x + y * y) - 1) / ((x * x + y * y) + 1); vec[0] = x * (1 + vec[2]); vec[1] = y * (1 + vec[2]); + // Transform the normal into the second crystal reference frame using + // the inverse misorientation (dgt). This is needed for the bicrystal + // symmetry computation below. vec2 = dgt * vec; - // Loop over all the symmetry operators in the given crystal symmetry + // Loop over all pairs of symmetry operators (O(nSym^2) per pixel). + // For each pair (sym1, sym2), we compute the symmetrically-equivalent + // misorientation and look up the GBCD bin for that misorientation + + // boundary-plane normal combination. for(int32 i = 0; i < nSym; i++) { - // get symmetry operator1 sym1 = m_OrientOps->getMatSymOpF(i); for(int32 j = 0; j < nSym; j++) { - // get symmetry operator2 sym2 = m_OrientOps->getMatSymOpF(j); sym2t = sym2.transpose(); - // calculate symmetric misorientation + // Compute the symmetrically-equivalent misorientation: + // dg2 = sym1 * dg * sym2^T + // This applies symmetry operator i on the left and j on the right. dg1 = dg * sym2t; dg2 = sym1 * dg1; @@ -252,6 +299,19 @@ const std::atomic_bool& ComputeGBCDPoleFigureDirect::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief In-core GBCD pole figure generation. + * + * Caches the entire GBCD array (all phases, all bins) into a local heap buffer, + * then uses ParallelData2DAlgorithm to compute pole figure pixel intensities in + * parallel. Each pixel's intensity is the symmetry-averaged GBCD value for the + * user-specified misorientation at the boundary-plane normal corresponding to + * that pixel's stereographic projection coordinate. + * + * The GBCD data, crystal structures, and output pole figure are all copied to/from + * local buffers via copyIntoBuffer()/copyFromBuffer() so that the parallel worker + * operates on plain raw pointers with no DataStore access. + */ Result<> ComputeGBCDPoleFigureDirect::operator()() { auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); @@ -259,39 +319,40 @@ Result<> ComputeGBCDPoleFigureDirect::operator()() DataPath cellIntensityArrayPath = m_InputValues->ImageGeometryPath.createChildPath(m_InputValues->CellAttributeMatrixName).createChildPath(m_InputValues->CellIntensityArrayName); auto& poleFigure = m_DataStructure.getDataRefAs(cellIntensityArrayPath); - // Cache entire GBCD array locally — this is the in-core (Direct) path - // where the full array fits in RAM. + // Cache the entire GBCD array into a contiguous local buffer. This is the + // in-core path: we expect the full array to fit in RAM. The buffer is passed + // as a raw float64* to the parallel worker, avoiding any DataStore access + // in the hot loop. const usize gbcdTotalElements = gbcd.getSize(); auto gbcdCache = std::make_unique(gbcdTotalElements); gbcd.getDataStoreRef().copyIntoBuffer(0, nonstd::span(gbcdCache.get(), gbcdTotalElements)); - // Cache crystal structures (ensemble-level, tiny) + // Cache ensemble-level crystal structures (typically < 10 elements). const usize numCrystalStructures = crystalStructures.getSize(); auto crystalStructuresCache = std::make_unique(numCrystalStructures); crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.get(), numCrystalStructures)); - // Cache pole figure output locally (300x300 = 90,000 elements, tiny) + // Allocate a local buffer for the output pole figure (e.g., 300x300 = 90,000 + // float64 elements). Initialize to zero; pixels outside the unit circle will + // remain at zero intensity. const usize poleFigureSize = poleFigure.getSize(); auto poleFigureCache = std::make_unique(poleFigureSize); std::fill(poleFigureCache.get(), poleFigureCache.get() + poleFigureSize, 0.0); + // ----- GBCD bin configuration ----- + // The GBCD is a 5-dimensional histogram. The 5 dimensions are: + // [0] misorientation phi1 (range: [0, pi/2]) + // [1] cos(misorientation Phi) (range: [0, 1]) + // [2] misorientation phi2 (range: [0, pi/2]) + // [3] boundary normal theta (range: [-sqrt(pi/2), sqrt(pi/2)], square-grid projection) + // [4] boundary normal phi (range: [-sqrt(pi/2), sqrt(pi/2)], square-grid projection) + // Each dimension is split into gbcdSizes[i] bins of width gbcdDeltas[i]. + // gbcdLimits[0..4] are the lower bounds and gbcdLimits[5..9] are the upper bounds. std::vector gbcdDeltas(5, 0); std::vector gbcdLimits(10, 0); std::vector gbcdSizes(5, 0); - // Original Ranges from Dave R. - // gbcdLimits[0] = 0.0f; - // gbcdLimits[1] = cosf(1.0f*m_pi); - // gbcdLimits[2] = 0.0f; - // gbcdLimits[3] = 0.0f; - // gbcdLimits[4] = cosf(1.0f*m_pi); - // gbcdLimits[5] = 2.0f*m_pi; - // gbcdLimits[6] = cosf(0.0f); - // gbcdLimits[7] = 2.0f*m_pi; - // gbcdLimits[8] = 2.0f*m_pi; - // gbcdLimits[9] = cosf(0.0f); - - // Greg R. Ranges + // Greg Rohrer's ranges for the 5D GBCD parameter space. gbcdLimits[0] = 0.0f; gbcdLimits[1] = 0.0f; gbcdLimits[2] = 0.0f; @@ -303,13 +364,15 @@ Result<> ComputeGBCDPoleFigureDirect::operator()() gbcdLimits[8] = 1.0f; gbcdLimits[9] = Constants::k_2PiD; - // reset the 3rd and 4th dimensions using the square grid approach + // Override the 3rd and 4th dimension bounds to use the Lambert equal-area + // square-grid projection instead of raw angular coordinates. gbcdLimits[3] = -sqrtf(Constants::k_PiOver2D); gbcdLimits[4] = -sqrtf(Constants::k_PiOver2D); gbcdLimits[8] = sqrtf(Constants::k_PiOver2D); gbcdLimits[9] = sqrtf(Constants::k_PiOver2D); - // get num components of GBCD + // Extract the 5D component shape from the GBCD DataArray. The GBCD array has + // one tuple per phase and a multi-dimensional component shape encoding the 5D bins. ShapeType cDims = gbcd.getComponentShape(); gbcdSizes[0] = static_cast(cDims[0]); @@ -318,13 +381,16 @@ Result<> ComputeGBCDPoleFigureDirect::operator()() gbcdSizes[3] = static_cast(cDims[3]); gbcdSizes[4] = static_cast(cDims[4]); + // Compute bin widths: delta = (upper_bound - lower_bound) / num_bins. gbcdDeltas[0] = (gbcdLimits[5] - gbcdLimits[0]) / static_cast(gbcdSizes[0]); gbcdDeltas[1] = (gbcdLimits[6] - gbcdLimits[1]) / static_cast(gbcdSizes[1]); gbcdDeltas[2] = (gbcdLimits[7] - gbcdLimits[2]) / static_cast(gbcdSizes[2]); gbcdDeltas[3] = (gbcdLimits[8] - gbcdLimits[3]) / static_cast(gbcdSizes[3]); gbcdDeltas[4] = (gbcdLimits[9] - gbcdLimits[4]) / static_cast(gbcdSizes[4]); - // Get our LaueOps pointer for the selected crystal structure + // Select the LaueOps instance for the phase of interest. The crystal structure + // enum (e.g., Cubic_High, Hexagonal_High) determines the set of symmetry + // operators used to enumerate equivalent misorientations. ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructuresCache[m_InputValues->PhaseOfInterest]]; int32 xPoints = m_InputValues->OutputImageDimension; @@ -336,14 +402,17 @@ Result<> ComputeGBCDPoleFigureDirect::operator()() m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Generating Intensity Plot for phase {}", m_InputValues->PhaseOfInterest)}); - // Use cached raw pointers — no OOC access in hot loop, parallelization is safe + // Launch multi-threaded pole figure computation. All data is accessed through + // locally-cached raw pointers (gbcdCache, poleFigureCache), so parallel writes + // to disjoint pixel regions are safe -- no DataStore access occurs in the hot loop. ParallelData2DAlgorithm dataAlg; dataAlg.setRange(0, xPoints, 0, yPoints); dataAlg.execute(ComputeGBCDPoleFigureImpl(poleFigureCache.get(), {xPoints, yPoints}, orientOps, gbcdDeltas, gbcdLimits, gbcdSizes, gbcdCache.get(), m_InputValues->PhaseOfInterest, m_InputValues->MisorientationRotation)); - // Write pole figure results back to the OOC store + // Write the computed pole figure intensities back to the DataStore. This is a + // single bulk write that works correctly for both in-core and OOC-backed stores. poleFigure.getDataStoreRef().copyFromBuffer(0, nonstd::span(poleFigureCache.get(), poleFigureSize)); return {}; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp index a2847945b3..795ce5b84b 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureDirect.hpp @@ -10,24 +10,62 @@ namespace nx::core { +/** + * @struct ComputeGBCDPoleFigureInputValues + * @brief Holds user-facing parameters for the GBCD Pole Figure computation. + * + * The Grain Boundary Character Distribution (GBCD) is a 5-dimensional histogram that + * captures the statistical distribution of grain boundary planes as a function of the + * misorientation between the two grains that meet at the boundary. This struct packages + * the parameters needed to extract a 2D stereographic projection (pole figure) from that + * 5D distribution for a specific misorientation and crystal phase. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureInputValues { - int32 PhaseOfInterest; - VectorFloat32Parameter::ValueType MisorientationRotation; - DataPath GBCDArrayPath; - DataPath CrystalStructuresArrayPath; - int32 OutputImageDimension; - DataPath ImageGeometryPath; - std::string CellAttributeMatrixName; - std::string CellIntensityArrayName; + int32 PhaseOfInterest; ///< 1-based phase index selecting which crystallographic phase's GBCD to visualize. + VectorFloat32Parameter::ValueType MisorientationRotation; ///< Misorientation as [angle(deg), axis_x, axis_y, axis_z]. The axis is normalized internally. + DataPath GBCDArrayPath; ///< Path to the Float64 GBCD array (tuples = phases, 5D component shape from ComputeGBCD). + DataPath CrystalStructuresArrayPath; ///< Path to the UInt32 ensemble array mapping phase ID -> EbsdLib crystal structure enum. + int32 OutputImageDimension; ///< Side length (pixels) of the square output pole figure image. + DataPath ImageGeometryPath; ///< Path to the output ImageGeometry that will hold the pole figure. + std::string CellAttributeMatrixName; ///< Name of the cell AttributeMatrix created under the output ImageGeometry. + std::string CellIntensityArrayName; ///< Name of the Float64 intensity array written into the cell AttributeMatrix. }; /** - * @class + * @class ComputeGBCDPoleFigureDirect + * @brief In-core (Direct) algorithm for generating a GBCD stereographic pole figure. + * + * This algorithm is selected by the dispatcher when the GBCD array resides entirely + * in contiguous in-memory storage. It performs the following steps: + * + * 1. Caches the entire GBCD array, crystal structures, and output pole figure into + * local heap buffers (the GBCD can be large -- millions of float64 elements -- + * but fits in RAM when the data is in-core). + * 2. For each pixel (x, y) in the output stereographic projection, computes the + * corresponding unit-sphere direction via inverse stereographic projection. + * 3. Loops over all pairs of crystal symmetry operators (nSym x nSym) and for each + * pair computes the symmetrically-equivalent misorientation. If the equivalent + * misorientation falls in the fundamental zone (all three Euler angles < pi/2), + * the 5D GBCD bin is looked up and accumulated. + * 4. The procedure is repeated in the second crystal reference frame (using the + * transpose of the misorientation matrix) to account for the bicrystal symmetry. + * 5. The accumulated sum is averaged over the count of valid symmetry pairs. + * 6. Uses ParallelData2DAlgorithm for multi-threaded computation across output pixels. + * 7. Writes the final pole figure back to the DataStore via copyFromBuffer(). + * + * @see ComputeGBCDPoleFigureScanline for the OOC-optimized variant. */ class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureDirect { public: + /** + * @brief Constructs the in-core GBCD pole figure algorithm. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic cancellation flag. + * @param inputValues Pointer to the shared parameter struct; must outlive this object. + */ ComputeGBCDPoleFigureDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeGBCDPoleFigureInputValues* inputValues); ~ComputeGBCDPoleFigureDirect() noexcept; @@ -36,15 +74,23 @@ class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureDirect ComputeGBCDPoleFigureDirect& operator=(const ComputeGBCDPoleFigureDirect&) = delete; ComputeGBCDPoleFigureDirect& operator=(ComputeGBCDPoleFigureDirect&&) noexcept = delete; + /** + * @brief Generates the pole figure using multi-threaded parallel pixel computation. + * @return Result<> (currently always succeeds). + */ Result<> operator()(); + /** + * @brief Returns the cancellation flag reference. + * @return const reference to the atomic cancellation flag. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeGBCDPoleFigureInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const ComputeGBCDPoleFigureInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp index e4226b2cc5..7638ca10fe 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.cpp @@ -17,18 +17,35 @@ using namespace nx::core; namespace { +/** + * @class ComputeGBCDPoleFigureImpl + * @brief Threaded worker for generating a GBCD stereographic pole figure (Scanline variant). + * + * This is the same pixel-computation logic as the Direct variant's worker. The only + * difference is in how the GBCD data is provided: + * - In the Direct variant, m_Gbcd points to the full multi-phase GBCD array, and + * m_PhaseOfInterest is used to offset into the correct phase slice. + * - In the Scanline variant, m_Gbcd points to a pre-extracted single-phase slice + * (already offset), and m_PhaseOfInterest is set to 0 so no additional offset + * is applied. + * + * This shared-worker design means the parallel pixel computation is identical regardless + * of whether the GBCD was loaded in full (Direct) or as a single-phase slice (Scanline). + * + * @see ComputeGBCDPoleFigureScanline::operator()() for the OOC data-loading strategy. + */ class ComputeGBCDPoleFigureImpl { private: - float64* m_PoleFigure; - std::array m_Dimensions; - ebsdlib::LaueOps::Pointer m_OrientOps; - const std::vector& m_GbcdDeltas; - const std::vector& m_GbcdLimits; - const std::vector& m_GbcdSizes; - const float64* m_Gbcd; - int32 m_PhaseOfInterest = 0; - const std::vector& m_MisorientationRotation; + float64* m_PoleFigure; ///< Output pole figure pixel intensities (xPoints * yPoints). + std::array m_Dimensions; ///< [xPoints, yPoints] of the output image. + ebsdlib::LaueOps::Pointer m_OrientOps; ///< LaueOps for the crystal structure of the phase of interest. + const std::vector& m_GbcdDeltas; ///< Bin width in each of the 5 GBCD dimensions. + const std::vector& m_GbcdLimits; ///< Lower [0-4] and upper [5-9] bounds for the 5 GBCD dimensions. + const std::vector& m_GbcdSizes; ///< Number of bins in each of the 5 GBCD dimensions. + const float64* m_Gbcd; ///< Pointer to the GBCD data (may be phase-offset or single-phase slice). + int32 m_PhaseOfInterest = 0; ///< Phase index offset (0 when using a pre-extracted phase slice). + const std::vector& m_MisorientationRotation; ///< User-specified misorientation [angle_deg, axis_x, axis_y, axis_z]. public: ComputeGBCDPoleFigureImpl(float64* poleFigurePtr, const std::array& dimensions, const ebsdlib::LaueOps::Pointer& orientOps, const std::vector& gbcdDeltasArray, @@ -225,6 +242,21 @@ const std::atomic_bool& ComputeGBCDPoleFigureScanline::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief OOC-optimized GBCD pole figure generation. + * + * The key difference from the Direct variant is the GBCD caching strategy: + * instead of caching the entire multi-phase GBCD array, this variant extracts + * only the single-phase slice needed for the requested PhaseOfInterest via a + * single copyIntoBuffer() call. This dramatically reduces memory consumption + * when the GBCD has many phases. + * + * Once the phase slice is cached locally, the computation is parallelized + * identically to the Direct path using ParallelData2DAlgorithm on the cached + * raw pointers. The m_PhaseOfInterest parameter is set to 0 when constructing + * the worker because the cached buffer already starts at the phase-of-interest + * offset. + */ Result<> ComputeGBCDPoleFigureScanline::operator()() { auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); @@ -232,21 +264,23 @@ Result<> ComputeGBCDPoleFigureScanline::operator()() DataPath cellIntensityArrayPath = m_InputValues->ImageGeometryPath.createChildPath(m_InputValues->CellAttributeMatrixName).createChildPath(m_InputValues->CellIntensityArrayName); auto& poleFigure = m_DataStructure.getDataRefAs(cellIntensityArrayPath); - // Cache crystal structures (ensemble-level, tiny) + // Cache ensemble-level crystal structures (typically < 10 elements). const usize numCrystalStructures = crystalStructures.getSize(); auto crystalStructuresCache = std::make_unique(numCrystalStructures); crystalStructures.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructuresCache.get(), numCrystalStructures)); - // Cache pole figure output locally (300x300 = 90,000 elements, tiny) + // Allocate a local buffer for the output pole figure. Initialize to zero; + // pixels outside the stereographic unit circle will remain at zero. const usize poleFigureSize = poleFigure.getSize(); auto poleFigureCache = std::make_unique(poleFigureSize); std::fill(poleFigureCache.get(), poleFigureCache.get() + poleFigureSize, 0.0); + // ----- GBCD bin configuration (same as Direct variant) ----- std::vector gbcdDeltas(5, 0); std::vector gbcdLimits(10, 0); std::vector gbcdSizes(5, 0); - // Greg R. Ranges + // Greg Rohrer's ranges for the 5D GBCD parameter space. gbcdLimits[0] = 0.0f; gbcdLimits[1] = 0.0f; gbcdLimits[2] = 0.0f; @@ -258,13 +292,14 @@ Result<> ComputeGBCDPoleFigureScanline::operator()() gbcdLimits[8] = 1.0f; gbcdLimits[9] = Constants::k_2PiD; - // reset the 3rd and 4th dimensions using the square grid approach + // Override the 3rd and 4th dimension bounds to use the Lambert equal-area + // square-grid projection. gbcdLimits[3] = -sqrtf(Constants::k_PiOver2D); gbcdLimits[4] = -sqrtf(Constants::k_PiOver2D); gbcdLimits[8] = sqrtf(Constants::k_PiOver2D); gbcdLimits[9] = sqrtf(Constants::k_PiOver2D); - // get num components of GBCD + // Extract the 5D component shape from the GBCD DataArray. ShapeType cDims = gbcd.getComponentShape(); gbcdSizes[0] = static_cast(cDims[0]); @@ -279,15 +314,20 @@ Result<> ComputeGBCDPoleFigureScanline::operator()() gbcdDeltas[3] = (gbcdLimits[8] - gbcdLimits[3]) / static_cast(gbcdSizes[3]); gbcdDeltas[4] = (gbcdLimits[9] - gbcdLimits[4]) / static_cast(gbcdSizes[4]); + // Total number of GBCD bins per phase (both hemispheres). int64 totalGbcdBins = gbcdSizes[0] * gbcdSizes[1] * gbcdSizes[2] * gbcdSizes[3] * gbcdSizes[4] * 2; - // OOC optimization: cache only the phase-of-interest slice of the GBCD array. - // One phase = totalGbcdBins elements, bounded by bin resolution not array size. + // ---- OOC optimization: cache only the single phase slice ---- + // The full GBCD array has (numPhases * totalGbcdBins) elements. For an OOC store, + // reading the entire array would load all phases' bins from disk. Instead, we + // compute the element offset for the phase-of-interest and read only that + // contiguous slice. This is the critical optimization: one phase's GBCD is + // typically 100K-500K float64 elements vs. millions for all phases combined. const usize phaseOffset = static_cast(m_InputValues->PhaseOfInterest) * static_cast(totalGbcdBins); auto gbcdPhaseCache = std::make_unique(static_cast(totalGbcdBins)); gbcd.getDataStoreRef().copyIntoBuffer(phaseOffset, nonstd::span(gbcdPhaseCache.get(), static_cast(totalGbcdBins))); - // Get our LaueOps pointer for the selected crystal structure + // Select the LaueOps instance for the phase of interest. ebsdlib::LaueOps::Pointer orientOps = ebsdlib::LaueOps::GetAllOrientationOps()[crystalStructuresCache[m_InputValues->PhaseOfInterest]]; int32 xPoints = m_InputValues->OutputImageDimension; @@ -295,14 +335,18 @@ Result<> ComputeGBCDPoleFigureScanline::operator()() m_MessageHandler({IFilter::Message::Type::Info, fmt::format("Generating Intensity Plot for phase {} (OOC)", m_InputValues->PhaseOfInterest)}); - // Parallel execution is safe — all data accessed through local cached buffers + // Parallel execution is safe because all data is accessed through locally-cached + // raw pointers -- no DataStore access occurs in the hot loop. ParallelData2DAlgorithm dataAlg; dataAlg.setRange(0, xPoints, 0, yPoints); - // Pass phaseOffset=0 since gbcdPhaseCache already points to the phase slice + // Pass phaseOfInterest=0 to the worker because gbcdPhaseCache already points to + // the start of the phase-of-interest slice. The worker's GBCD indexing formula + // uses (phaseOfInterest * totalGbcdBins) as an offset, so passing 0 means no + // additional offset is applied to our already-offset buffer. dataAlg.execute(ComputeGBCDPoleFigureImpl(poleFigureCache.get(), {xPoints, yPoints}, orientOps, gbcdDeltas, gbcdLimits, gbcdSizes, gbcdPhaseCache.get(), 0, m_InputValues->MisorientationRotation)); - // Write pole figure results back to the OOC store + // Write the computed pole figure intensities back to the DataStore. poleFigure.getDataStoreRef().copyFromBuffer(0, nonstd::span(poleFigureCache.get(), poleFigureSize)); return {}; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp index 3945dd1d34..0fafa00556 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeGBCDPoleFigureScanline.hpp @@ -14,13 +14,40 @@ struct ComputeGBCDPoleFigureInputValues; /** * @class ComputeGBCDPoleFigureScanline - * @brief OOC-safe variant that accesses the GBCD array through the DataStore - * without caching the entire array in RAM. Runs single-threaded since - * DataStore per-element access is not thread-safe for concurrent reads. + * @brief Out-of-core (Scanline) algorithm for generating a GBCD stereographic pole figure. + * + * This algorithm is selected by the dispatcher when the GBCD array is backed by + * chunked (OOC) storage on disk. The full GBCD array can be very large (millions of + * float64 elements across all phases), but for a single pole figure only the bins + * belonging to one phase are accessed. + * + * **OOC optimization**: Instead of caching the entire GBCD array (as the Direct + * variant does), this algorithm caches only the single-phase slice of the GBCD via + * a single copyIntoBuffer() call. For a typical GBCD with 5D bin resolution, one + * phase slice is on the order of hundreds of thousands of float64 elements -- far + * smaller than the full multi-phase array. This dramatically reduces the memory + * footprint and avoids random-access chunk thrashing across phase boundaries. + * + * Once the phase slice is cached in a local buffer, the actual pole-figure + * computation is identical to the Direct variant and runs multi-threaded using + * ParallelData2DAlgorithm on the cached raw pointers. + * + * **Memory footprint**: O(totalGbcdBins) for one phase, plus O(outputDim^2) for + * the pole figure cache. Both are bounded by the GBCD bin resolution, not by the + * total number of phases. + * + * @see ComputeGBCDPoleFigureDirect for the in-core variant that caches the full GBCD. */ class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureScanline { public: + /** + * @brief Constructs the OOC GBCD pole figure algorithm. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic cancellation flag. + * @param inputValues Pointer to the shared parameter struct; must outlive this object. + */ ComputeGBCDPoleFigureScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeGBCDPoleFigureInputValues* inputValues); ~ComputeGBCDPoleFigureScanline() noexcept; @@ -29,15 +56,23 @@ class ORIENTATIONANALYSIS_EXPORT ComputeGBCDPoleFigureScanline ComputeGBCDPoleFigureScanline& operator=(const ComputeGBCDPoleFigureScanline&) = delete; ComputeGBCDPoleFigureScanline& operator=(ComputeGBCDPoleFigureScanline&&) noexcept = delete; + /** + * @brief Generates the pole figure by caching only the phase-of-interest GBCD slice. + * @return Result<> (currently always succeeds). + */ Result<> operator()(); + /** + * @brief Returns the cancellation flag reference. + * @return const reference to the atomic cancellation flag. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeGBCDPoleFigureInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const ComputeGBCDPoleFigureInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp index ff4e8d4988..59e1571ce6 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.cpp @@ -21,8 +21,20 @@ ComputeIPFColors::ComputeIPFColors(DataStructure& dataStructure, const IFilter:: ComputeIPFColors::~ComputeIPFColors() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches IPF color computation to the appropriate algorithm based on + * storage type. + * + * The three arrays checked for OOC status are the Euler angles (input, 3-component + * float32), the cell phases (input, 1-component int32), and the IPF colors (output, + * 3-component uint8). If any of them are backed by chunked OOC storage, the Scanline + * path is selected to avoid chunk thrashing; otherwise the parallel Direct path is used. + */ Result<> ComputeIPFColors::operator()() { + // Retrieve raw IDataArray pointers for storage-type inspection by DispatchAlgorithm. + // These are only used for the AnyOutOfCore() check -- the actual typed access + // happens inside ComputeIPFColorsDirect or ComputeIPFColorsScanline. auto* eulersArray = m_DataStructure.getDataAs(m_InputValues->cellEulerAnglesArrayPath); auto* phasesArray = m_DataStructure.getDataAs(m_InputValues->cellPhasesArrayPath); auto* ipfColorsArray = m_DataStructure.getDataAs(m_InputValues->cellIpfColorsArrayPath); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp index f72cf32f5b..0e88137a3b 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColors.hpp @@ -11,20 +11,51 @@ namespace nx::core { +/** + * @struct ComputeIPFColorsInputValues + * @brief Holds the user-facing parameters for the Compute IPF Colors algorithm. + * + * Inverse Pole Figure (IPF) colors map a crystal orientation to a position on the + * unit stereographic triangle for the crystal's Laue class, producing a unique RGB + * color. The reference direction determines which sample axis is projected into the + * crystal frame before the color lookup. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsInputValues { - std::vector referenceDirection; - bool useGoodVoxels = false; - DataPath goodVoxelsArrayPath; - DataPath cellPhasesArrayPath; - DataPath cellEulerAnglesArrayPath; - DataPath crystalStructuresArrayPath; - DataPath cellIpfColorsArrayPath; + std::vector referenceDirection; ///< Sample-frame reference direction (typically [0,0,1]) that is projected into the crystal frame for IPF color computation. + bool useGoodVoxels = false; ///< When true, voxels whose mask value is false/0 are colored black (skipped). + DataPath goodVoxelsArrayPath; ///< Path to the boolean or uint8 mask array. Only used when useGoodVoxels is true. + DataPath cellPhasesArrayPath; ///< Path to the Int32 array of per-voxel phase IDs (1-based; 0 = unindexed). + DataPath cellEulerAnglesArrayPath; ///< Path to the Float32 array of Euler angles (phi1, Phi, phi2) in radians, 3 components per tuple. + DataPath crystalStructuresArrayPath; ///< Path to the UInt32 ensemble array mapping phase ID -> EbsdLib crystal structure enum. + DataPath cellIpfColorsArrayPath; ///< Path to the output UInt8 array of RGB colors, 3 components per tuple. }; +/** + * @class ComputeIPFColors + * @brief Dispatcher that selects between in-core and out-of-core IPF color algorithms. + * + * This class serves as the entry point called by ComputeIPFColorsFilter::executeImpl(). + * It inspects the backing storage of the Euler-angle, phase, and IPF-color arrays using + * DispatchAlgorithm: + * + * - **In-core (ComputeIPFColorsDirect)**: Uses ParallelDataAlgorithm for multi-threaded + * random access when all arrays reside in contiguous RAM. + * - **Out-of-core (ComputeIPFColorsScanline)**: Reads and writes data in fixed-size + * chunks via copyIntoBuffer()/copyFromBuffer() to avoid OOC chunk thrashing. + * + * @see ComputeIPFColorsDirect, ComputeIPFColorsScanline, DispatchAlgorithm + */ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColors { public: + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all input and output arrays. + * @param msgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic flag checked periodically to support user cancellation. + * @param inputValues Pointer to the parameter struct; must outlive this object. + */ ComputeIPFColors(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ComputeIPFColorsInputValues* inputValues); ~ComputeIPFColors() noexcept; @@ -33,13 +64,18 @@ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColors ComputeIPFColors& operator=(const ComputeIPFColors&) = delete; ComputeIPFColors& operator=(ComputeIPFColors&&) = delete; + /** + * @brief Dispatches to ComputeIPFColorsDirect or ComputeIPFColorsScanline based + * on whether any of the involved arrays use out-of-core storage. + * @return Result<> with any errors (e.g., phase mismatch warnings). + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const IFilter::MessageHandler& m_MessageHandler; - const std::atomic_bool& m_ShouldCancel; - const ComputeIPFColorsInputValues* m_InputValues = nullptr; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const ComputeIPFColorsInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp index 1e9bac69fc..ba12d7c443 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.cpp @@ -17,8 +17,16 @@ namespace { /** - * @brief The ComputeIPFColorsImpl class implements a threaded algorithm that computes the IPF - * colors for each element in a geometry + * @class ComputeIPFColorsImpl + * @brief Threaded worker for computing IPF colors on a per-voxel range. + * + * Each instance is invoked by ParallelDataAlgorithm on a disjoint [start, end) tuple + * range. The worker reads Euler angles, looks up the crystal symmetry for the voxel's + * phase, and calls EbsdLib's LaueOps::generateIPFColor() to map the reference direction + * into the crystal frame and obtain an RGB color on the inverse pole figure triangle. + * + * This worker holds AbstractDataStore references, which is safe ONLY when all stores + * are in-core (contiguous memory). For OOC-backed stores, see ComputeIPFColorsScanline. */ class ComputeIPFColorsImpl { @@ -38,6 +46,18 @@ class ComputeIPFColorsImpl virtual ~ComputeIPFColorsImpl() = default; + /** + * @brief Computes IPF colors for voxels in the range [start, end). + * + * Templated on the mask element type (bool or uint8) because the mask array + * can be either DataType::boolean or DataType::uint8. When no mask is provided, + * the bool specialization is used with a nullptr maskArray, which causes all + * voxels to be computed. + * + * @tparam T Element type of the mask array (bool or uint8). + * @param start First tuple index (inclusive). + * @param end Last tuple index (exclusive). + */ template void convert(size_t start, size_t end) const { @@ -48,6 +68,8 @@ class ComputeIPFColorsImpl maskArray = dynamic_cast(m_GoodVoxels); } + // Create thread-local copies of LaueOps to avoid sharing mutable state. + // Each LaueOps instance encapsulates the symmetry operators for one Laue class. std::vector ops = ebsdlib::LaueOps::GetAllOrientationOps(); std::array refDir = {m_ReferenceDir[0], m_ReferenceDir[1], m_ReferenceDir[2]}; std::array dEuler = {0.0, 0.0, 0.0}; @@ -62,28 +84,42 @@ class ComputeIPFColorsImpl return; } phase = m_CellPhases[i]; + // Each voxel's color is stored as 3 consecutive uint8 values (R, G, B). + // Default to black (0, 0, 0); overwritten only if orientation is valid. index = i * 3; m_CellIPFColors.setValue(index, 0); m_CellIPFColors.setValue(index + 1, 0); m_CellIPFColors.setValue(index + 2, 0); + + // Read the three Euler angles (phi1, Phi, phi2) in radians for this voxel. dEuler[0] = m_CellEulerAngles.getValue(index); dEuler[1] = m_CellEulerAngles.getValue(index + 1); dEuler[2] = m_CellEulerAngles.getValue(index + 2); - // Make sure we are using a valid Euler Angles with valid crystal symmetry + // If a mask is active, skip voxels marked as bad (mask == false/0). calcIPF = true; if(nullptr != maskArray) { calcIPF = (*maskArray)[i]; } - // Sanity check the phase data to make sure we do not walk off the end of the array + // Guard against phase IDs that exceed the crystal structures array length. + // This indicates corrupt input data; we count occurrences for a post-run warning. if(phase >= m_NumPhases) { m_Filter->incrementPhaseWarningCount(); } + // Compute the IPF color only if: + // 1. Phase ID is within the ensemble array bounds. + // 2. The mask allows computation (or no mask is used). + // 3. The crystal structure is a recognized Laue group (not Unknown). if(phase < m_NumPhases && calcIPF && m_CrystalStructures[phase] < ebsdlib::CrystalStructure::LaueGroupEnd) { + // generateIPFColor() transforms refDir into the crystal frame using the + // orientation defined by the Euler angles, then maps the resulting crystal + // direction to an RGB color on the stereographic triangle for the crystal's + // Laue class. The 'false' parameter disables the conversion from radians + // (our Euler angles are already in radians). argb = ops[m_CrystalStructures[phase]]->generateIPFColor(dEuler.data(), refDir.data(), false); m_CellIPFColors.setValue(index, static_cast(nx::core::RgbColor::dRed(argb))); m_CellIPFColors.setValue(index + 1, static_cast(nx::core::RgbColor::dGreen(argb))); @@ -92,6 +128,10 @@ class ComputeIPFColorsImpl } } + /** + * @brief Dispatches to the correct convert() instantiation based on the + * runtime DataType of the mask array. + */ void run(size_t start, size_t end) const { if(m_GoodVoxels != nullptr) @@ -107,24 +147,29 @@ class ComputeIPFColorsImpl } else { + // No mask provided -- compute IPF color for every voxel. convert(start, end); } } + /** + * @brief ParallelDataAlgorithm entry point. Called once per thread with a + * disjoint Range of tuple indices. + */ void operator()(const Range& range) const { run(range.min(), range.max()); } private: - ComputeIPFColorsDirect* m_Filter = nullptr; - nx::core::FloatVec3 m_ReferenceDir; - nx::core::Float32AbstractDataStore& m_CellEulerAngles; - nx::core::Int32AbstractDataStore& m_CellPhases; - nx::core::UInt32AbstractDataStore& m_CrystalStructures; - int32_t m_NumPhases = 0; - const nx::core::IDataArray* m_GoodVoxels = nullptr; - nx::core::UInt8AbstractDataStore& m_CellIPFColors; + ComputeIPFColorsDirect* m_Filter = nullptr; ///< Back-pointer for cancellation and phase-warning accumulation. + nx::core::FloatVec3 m_ReferenceDir; ///< Normalized sample-frame reference direction. + nx::core::Float32AbstractDataStore& m_CellEulerAngles; ///< DataStore of Euler angles (3 components per tuple, radians). + nx::core::Int32AbstractDataStore& m_CellPhases; ///< DataStore of per-voxel phase IDs. + nx::core::UInt32AbstractDataStore& m_CrystalStructures; ///< DataStore of ensemble crystal structure enums. + int32_t m_NumPhases = 0; ///< Number of phases in the ensemble (bounds check for phase IDs). + const nx::core::IDataArray* m_GoodVoxels = nullptr; ///< Optional mask array (bool or uint8). nullptr means compute all. + nx::core::UInt8AbstractDataStore& m_CellIPFColors; ///< Output DataStore of RGB colors (3 components per tuple). }; } // namespace @@ -142,26 +187,42 @@ ComputeIPFColorsDirect::ComputeIPFColorsDirect(DataStructure& dataStructure, con ComputeIPFColorsDirect::~ComputeIPFColorsDirect() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief In-core IPF color computation using multi-threaded ParallelDataAlgorithm. + * + * All data arrays are accessed through AbstractDataStore references, which provide + * O(1) random access when the backing store is in-memory. The ParallelDataAlgorithm + * splits the total voxel count into sub-ranges and dispatches ComputeIPFColorsImpl + * workers to separate threads. + * + * requireArraysInMemory() is called to pin all arrays in RAM for the duration of + * parallel execution, preventing potential issues if a store implements lazy loading. + */ Result<> ComputeIPFColorsDirect::operator()() { std::vector orientationOps = ebsdlib::LaueOps::GetAllOrientationOps(); + // Retrieve typed array references. These are guaranteed to exist because + // the filter's preflightImpl() validated them via selection parameters. nx::core::Float32Array& eulers = m_DataStructure.getDataRefAs(m_InputValues->cellEulerAnglesArrayPath); nx::core::Int32Array& phases = m_DataStructure.getDataRefAs(m_InputValues->cellPhasesArrayPath); - nx::core::UInt32Array& crystalStructures = m_DataStructure.getDataRefAs(m_InputValues->crystalStructuresArrayPath); - nx::core::UInt8Array& ipfColors = m_DataStructure.getDataRefAs(m_InputValues->cellIpfColorsArrayPath); m_PhaseWarningCount = 0; size_t totalPoints = eulers.getNumberOfTuples(); + // The crystal structures array has one entry per phase (including phase 0 = unknown). + // numPhases is used as an upper bound for phase ID validation. int32_t numPhases = static_cast(crystalStructures.getNumberOfTuples()); - // Make sure we are dealing with a unit 1 vector. - nx::core::FloatVec3 normRefDir = m_InputValues->referenceDirection; // Make a copy of the reference Direction + // Normalize the reference direction to a unit vector. The user may supply + // any non-zero vector (e.g., [0,0,1] for the sample Z axis). + nx::core::FloatVec3 normRefDir = m_InputValues->referenceDirection; normRefDir = normRefDir.normalize(); + // Collect all arrays that will be accessed by the parallel workers so that + // ParallelDataAlgorithm can pin them in memory for the duration of execution. typename IParallelAlgorithm::AlgorithmArrays algArrays; algArrays.push_back(&eulers); algArrays.push_back(&phases); @@ -175,13 +236,16 @@ Result<> ComputeIPFColorsDirect::operator()() algArrays.push_back(goodVoxelsArray); } - // Allow data-based parallelization + // Launch multi-threaded execution. Each thread processes a contiguous sub-range + // of tuple indices via ComputeIPFColorsImpl::operator()(Range). ParallelDataAlgorithm dataAlg; dataAlg.setRange(0, totalPoints); dataAlg.requireArraysInMemory(algArrays); dataAlg.execute(ComputeIPFColorsImpl(this, normRefDir, eulers, phases, crystalStructures, numPhases, goodVoxelsArray, ipfColors)); + // After all threads have joined, check whether any voxels had phase IDs that + // exceeded the ensemble array bounds. This is a data-quality issue in the input. if(m_PhaseWarningCount > 0) { std::string message = fmt::format("The Ensemble Phase information only references {} phase(s) but {} cell(s) had a phase value greater than {}. \ diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp index b17a926628..821a3832ee 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsDirect.hpp @@ -11,9 +11,41 @@ namespace nx::core struct ComputeIPFColorsInputValues; +/** + * @class ComputeIPFColorsDirect + * @brief In-core (Direct) algorithm for computing Inverse Pole Figure colors. + * + * This algorithm is selected by the dispatcher when all relevant arrays reside + * in contiguous in-memory DataStores. It uses ParallelDataAlgorithm to split + * the voxel range across threads, with each thread computing IPF colors for its + * assigned range by: + * + * 1. Reading Euler angles (phi1, Phi, phi2) for the voxel. + * 2. Checking the phase ID against the crystal-structure ensemble array. + * 3. Calling LaueOps::generateIPFColor() to transform the user-specified + * reference direction into the crystal frame and map it to an RGB color + * on the Laue-class-specific inverse pole figure triangle. + * + * An optional mask array allows pre-indexed voxels to be skipped (colored black). + * + * **Thread safety**: The parallel worker (ComputeIPFColorsImpl) accesses + * AbstractDataStore references, which requires ParallelDataAlgorithm's + * requireArraysInMemory() to lock the arrays in RAM for the duration of + * execution. This is safe for in-core DataStores but would be dangerous for + * OOC-backed stores. + * + * @see ComputeIPFColorsScanline for the OOC-optimized variant. + */ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsDirect { public: + /** + * @brief Constructs the in-core IPF color algorithm. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param msgHandler Message handler for progress/warning messages. + * @param shouldCancel Atomic cancellation flag checked inside the parallel worker. + * @param inputValues Pointer to the shared parameter struct; must outlive this object. + */ ComputeIPFColorsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, const ComputeIPFColorsInputValues* inputValues); ~ComputeIPFColorsDirect() noexcept; @@ -22,17 +54,34 @@ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsDirect ComputeIPFColorsDirect& operator=(const ComputeIPFColorsDirect&) = delete; ComputeIPFColorsDirect& operator=(ComputeIPFColorsDirect&&) = delete; + /** + * @brief Computes IPF colors for all voxels using multi-threaded random access. + * @return Result<> with an error if phase data is inconsistent. + */ Result<> operator()(); + /** + * @brief Thread-safe increment of the phase-mismatch warning counter. + * + * Called from parallel worker threads when a voxel's phase ID exceeds the + * number of entries in the crystal structures ensemble array. The count is + * not atomic (minor race is acceptable since it is only used for a warning + * message), but incrementing is trivial and the final check is post-join. + */ void incrementPhaseWarningCount(); + + /** + * @brief Returns the current cancellation state. + * @return true if the user has requested cancellation. + */ bool shouldCancel() const; private: - DataStructure& m_DataStructure; - const IFilter::MessageHandler& m_MessageHandler; - const std::atomic_bool& m_ShouldCancel; - const ComputeIPFColorsInputValues* m_InputValues = nullptr; - int32_t m_PhaseWarningCount = 0; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const ComputeIPFColorsInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. + int32_t m_PhaseWarningCount = 0; ///< Accumulates the number of voxels with out-of-range phase IDs. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp index dfde285e6a..54baed146c 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.cpp @@ -17,6 +17,15 @@ using namespace nx::core; namespace { +/** + * @brief Number of tuples processed per chunk in the scanline loop. + * + * 65,536 tuples is chosen as a balance between minimizing the number of + * copyIntoBuffer()/copyFromBuffer() round-trips and keeping the per-chunk + * heap allocation small (~768 KB for 3-component float32 Euler angles). + * This value does NOT need to align with the OOC chunk size; the DataStore's + * bulk I/O methods handle partial and cross-chunk reads internally. + */ constexpr usize k_ChunkTuples = 65536; } // namespace @@ -34,8 +43,25 @@ ComputeIPFColorsScanline::ComputeIPFColorsScanline(DataStructure& dataStructure, ComputeIPFColorsScanline::~ComputeIPFColorsScanline() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief OOC-safe IPF color computation using sequential chunk-based bulk I/O. + * + * The algorithm processes the entire tuple range in fixed-size chunks of + * k_ChunkTuples. For each chunk: + * 1. Bulk-read Euler angles, phase IDs, and (optionally) mask values from + * the OOC-backed DataStores into local heap buffers. + * 2. Compute IPF colors for every tuple in the chunk (same LaueOps logic + * as the Direct path). + * 3. Bulk-write the computed RGB colors back to the output DataStore. + * + * This linear access pattern ensures that each OOC disk chunk is read at most + * once per pass, avoiding the random-access chunk thrashing that would occur + * if ParallelDataAlgorithm workers accessed the OOC stores concurrently. + */ Result<> ComputeIPFColorsScanline::operator()() { + // Create all LaueOps instances upfront. The vector is indexed by crystal + // structure enum (e.g., Cubic_High = 1, Hexagonal_High = 2, etc.). std::vector ops = ebsdlib::LaueOps::GetAllOrientationOps(); auto& eulers = m_DataStructure.getDataRefAs(m_InputValues->cellEulerAnglesArrayPath); @@ -46,32 +72,42 @@ Result<> ComputeIPFColorsScanline::operator()() const usize totalPoints = eulers.getNumberOfTuples(); const int32 numPhases = static_cast(crystalStructuresArray.getNumberOfTuples()); - // Cache crystal structures locally (ensemble-level, tiny) + // Cache crystal structures locally. This ensemble-level array is tiny (one entry + // per phase) so it is always worth copying into a contiguous local vector to + // avoid per-element OOC access during the inner loop. std::vector crystalStructures(numPhases); crystalStructuresArray.getDataStoreRef().copyIntoBuffer(0, nonstd::span(crystalStructures.data(), static_cast(numPhases))); - // Normalize reference direction + // Normalize the reference direction to a unit vector and promote to double + // for compatibility with EbsdLib's generateIPFColor() API. FloatVec3 normRefDir = m_InputValues->referenceDirection; normRefDir = normRefDir.normalize(); std::array refDir = {normRefDir[0], normRefDir[1], normRefDir[2]}; - // Optional mask array + // Optional mask array -- only retrieved if the user opted into masking. const IDataArray* goodVoxelsArray = nullptr; if(m_InputValues->useGoodVoxels) { goodVoxelsArray = m_DataStructure.getDataAs(m_InputValues->goodVoxelsArrayPath); } + // Obtain DataStore references for bulk I/O. We never use operator[] on these + // stores; all access goes through copyIntoBuffer()/copyFromBuffer(). const auto& eulersStore = eulers.getDataStoreRef(); const auto& phasesStore = phases.getDataStoreRef(); auto& ipfColorsStore = ipfColors.getDataStoreRef(); - // Allocate chunk buffers + // Allocate fixed-size chunk buffers on the heap. These persist across chunks + // to avoid repeated allocation/deallocation. + // eulerBuf: k_ChunkTuples * 3 float32 (~768 KB) + // phasesBuf: k_ChunkTuples * 1 int32 (~256 KB) + // colorBuf: k_ChunkTuples * 3 uint8 (~192 KB) auto eulerBuf = std::make_unique(k_ChunkTuples * 3); auto phasesBuf = std::make_unique(k_ChunkTuples); auto colorBuf = std::make_unique(k_ChunkTuples * 3); - // Mask buffer — only allocated if needed + // Mask buffer -- only allocated for the mask type that is actually in use, + // to avoid wasting memory when no mask is needed. std::unique_ptr boolMaskBuf; std::unique_ptr uint8MaskBuf; const bool hasBoolMask = goodVoxelsArray != nullptr && goodVoxelsArray->getDataType() == DataType::boolean; @@ -88,6 +124,7 @@ Result<> ComputeIPFColorsScanline::operator()() int32 phaseWarningCount = 0; std::array dEuler = {0.0, 0.0, 0.0}; + // ---- Main scanline loop: process k_ChunkTuples tuples per iteration ---- for(usize offset = 0; offset < totalPoints; offset += k_ChunkTuples) { if(m_ShouldCancel) @@ -95,13 +132,19 @@ Result<> ComputeIPFColorsScanline::operator()() return {}; } + // The last chunk may be smaller than k_ChunkTuples. const usize count = std::min(k_ChunkTuples, totalPoints - offset); - // Bulk read cell-level data + // Bulk-read cell-level data for this chunk. The copyIntoBuffer() calls + // issue sequential reads against the underlying OOC store, which reads + // disk chunks in order and avoids thrashing. + // Note: Euler angles have 3 components per tuple, so the element offset + // and span size are multiplied by 3. eulersStore.copyIntoBuffer(offset * 3, nonstd::span(eulerBuf.get(), count * 3)); phasesStore.copyIntoBuffer(offset, nonstd::span(phasesBuf.get(), count)); - // Read mask chunk if applicable + // Read the mask chunk if applicable. The mask type is determined once + // before the loop and the appropriate buffer is used. if(hasBoolMask) { dynamic_cast(goodVoxelsArray)->getDataStoreRef().copyIntoBuffer(offset, nonstd::span(boolMaskBuf.get(), count)); @@ -111,11 +154,12 @@ Result<> ComputeIPFColorsScanline::operator()() dynamic_cast(goodVoxelsArray)->getDataStoreRef().copyIntoBuffer(offset, nonstd::span(uint8MaskBuf.get(), count)); } - // Process chunk + // Process every tuple in the chunk. This inner loop operates entirely on + // local heap buffers with no OOC store access -- all the I/O happened above. for(usize i = 0; i < count; i++) { const usize ci = i * 3; - // Default to black + // Default color is black (R=0, G=0, B=0) for bad/unindexed voxels. colorBuf[ci] = 0; colorBuf[ci + 1] = 0; colorBuf[ci + 2] = 0; @@ -125,6 +169,7 @@ Result<> ComputeIPFColorsScanline::operator()() dEuler[1] = eulerBuf[ci + 1]; dEuler[2] = eulerBuf[ci + 2]; + // Apply mask: skip voxels where the mask indicates bad data. bool calcIPF = true; if(hasBoolMask) { @@ -140,6 +185,10 @@ Result<> ComputeIPFColorsScanline::operator()() phaseWarningCount++; } + // Compute the IPF color only for valid, unmasked voxels with a recognized + // crystal structure. The generateIPFColor() call is the same as in the + // Direct path -- the only difference is that the data was pre-loaded from + // an OOC store via bulk I/O instead of accessed through random AbstractDataStore calls. if(phase < numPhases && calcIPF && crystalStructures[phase] < ebsdlib::CrystalStructure::LaueGroupEnd) { Rgba argb = ops[crystalStructures[phase]]->generateIPFColor(dEuler.data(), refDir.data(), false); @@ -149,7 +198,8 @@ Result<> ComputeIPFColorsScanline::operator()() } } - // Bulk write colors + // Bulk-write the computed colors back to the output store. Like the reads, + // this is a sequential write that aligns with the OOC chunk layout. ipfColorsStore.copyFromBuffer(offset * 3, nonstd::span(colorBuf.get(), count * 3)); } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp index ba633a8af2..96170950d6 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeIPFColorsScanline.hpp @@ -11,9 +11,47 @@ namespace nx::core struct ComputeIPFColorsInputValues; +/** + * @class ComputeIPFColorsScanline + * @brief Out-of-core (Scanline) algorithm for computing Inverse Pole Figure colors. + * + * This algorithm is selected by the dispatcher when any of the Euler-angle, phase, + * or IPF-color arrays are backed by chunked (OOC) storage on disk. It avoids the + * random-access pattern of the in-core path, which would trigger expensive + * chunk load/evict cycles ("chunk thrashing") when data does not fit in RAM. + * + * **Strategy**: The algorithm processes voxels in sequential, fixed-size chunks + * of k_ChunkTuples (65,536) tuples at a time: + * + * 1. Bulk-read Euler angles, phase IDs, and (optionally) the mask for the + * current chunk into local heap buffers via copyIntoBuffer(). + * 2. Compute IPF colors for every voxel in the chunk using the same + * LaueOps::generateIPFColor() logic as the in-core path. + * 3. Bulk-write the computed RGB colors back via copyFromBuffer(). + * + * Because OOC stores are chunked contiguously along the tuple dimension, this + * linear access pattern reads each disk chunk at most once, achieving throughput + * limited only by sequential disk I/O rather than random-access latency. + * + * **Single-threaded**: The algorithm runs single-threaded because the chunk + * buffers are shared state and because OOC disk I/O does not benefit from + * multi-threaded access to the same store. + * + * **Memory footprint**: O(k_ChunkTuples) per array -- roughly 1 MB total for + * the default chunk size, regardless of dataset size. + * + * @see ComputeIPFColorsDirect for the multi-threaded in-core variant. + */ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsScanline { public: + /** + * @brief Constructs the OOC IPF color algorithm. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param msgHandler Message handler for progress/info messages. + * @param shouldCancel Atomic cancellation flag checked once per chunk. + * @param inputValues Pointer to the shared parameter struct; must outlive this object. + */ ComputeIPFColorsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, const ComputeIPFColorsInputValues* inputValues); ~ComputeIPFColorsScanline() noexcept; @@ -22,13 +60,17 @@ class ORIENTATIONANALYSIS_EXPORT ComputeIPFColorsScanline ComputeIPFColorsScanline& operator=(const ComputeIPFColorsScanline&) = delete; ComputeIPFColorsScanline& operator=(ComputeIPFColorsScanline&&) = delete; + /** + * @brief Computes IPF colors for all voxels using sequential chunk-based I/O. + * @return Result<> with an error if phase data is inconsistent. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const IFilter::MessageHandler& m_MessageHandler; - const std::atomic_bool& m_ShouldCancel; - const ComputeIPFColorsInputValues* m_InputValues = nullptr; + DataStructure& m_DataStructure; ///< Reference to the live DataStructure. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for user-facing messages. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const ComputeIPFColorsInputValues* m_InputValues = nullptr; ///< Borrowed pointer to input parameters. }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp index 13b221c34e..ad80b56173 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.cpp @@ -24,6 +24,19 @@ ComputeKernelAvgMisorientations::ComputeKernelAvgMisorientations(DataStructure& ComputeKernelAvgMisorientations::~ComputeKernelAvgMisorientations() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Computes the average misorientation between each voxel and its + * neighbors within a user-specified kernel (kX x kY x kZ). Only neighbors + * belonging to the same feature are included in the average. + * + * OOC strategy: Replaced the parallel 3D range-based approach with Z-plane + * slab-based sequential processing. For each Z-plane, a slab spanning + * [plane - kZ, plane + kZ] is bulk-read via copyIntoBuffer, covering all + * neighbor lookups for voxels on that plane. The output for each plane is + * bulk-written via copyFromBuffer. This converts random 3D neighbor access + * (which caused severe OOC chunk thrashing) into bounded sequential reads + * proportional to (2*kZ+1) slices per plane. + */ Result<> ComputeKernelAvgMisorientations::operator()() { auto* gridGeom = m_DataStructure.getDataAs(m_InputValues->InputImageGeometry); @@ -38,13 +51,14 @@ Result<> ComputeKernelAvgMisorientations::operator()() const int32 kY = kernelSize[1]; const int32 kX = kernelSize[0]; - // Get DataStore references for bulk I/O + // DataStore references for bulk I/O — all cell-level reads/writes use + // copyIntoBuffer/copyFromBuffer to avoid per-element OOC virtual dispatch. const auto& cellPhasesStore = m_DataStructure.getDataRefAs(m_InputValues->CellPhasesArrayPath).getDataStoreRef(); const auto& featureIdsStore = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath).getDataStoreRef(); const auto& quatsStore = m_DataStructure.getDataRefAs(m_InputValues->QuatsArrayPath).getDataStoreRef(); auto& outputStore = m_DataStructure.getDataRefAs(m_InputValues->KernelAverageMisorientationsArrayName).getDataStoreRef(); - // Cache ensemble-level crystalStructures locally (tiny array, avoids per-element OOC overhead) + // Bulk-read ensemble-level crystal structures into local memory. const auto& crystalStructuresStore = m_DataStructure.getDataRefAs(m_InputValues->CrystalStructuresArrayPath).getDataStoreRef(); const usize numCrystalStructures = crystalStructuresStore.getNumberOfTuples(); std::vector crystalStructuresLocal(numCrystalStructures); @@ -65,13 +79,15 @@ Result<> ComputeKernelAvgMisorientations::operator()() { break; } - // Compute slab Z range (clamped to volume bounds) + // Compute the Z-extent of the slab needed for this plane's kernel lookups. + // The slab covers [plane - kZ, plane + kZ], clamped to volume bounds. const int64 slabZMin = std::max(static_cast(0), plane - kZ); const int64 slabZMax = std::min(zPoints - 1, plane + kZ); const usize slabZCount = static_cast(slabZMax - slabZMin + 1); const usize slabTuples = slabZCount * sliceSize; - // Read slab data via copyIntoBuffer (OOC-safe bulk I/O) + // Bulk-read the entire slab via copyIntoBuffer. This is the key OOC + // optimization: a single contiguous read replaces many random accesses. const usize slabStartTuple = static_cast(slabZMin) * sliceSize; std::vector slabFeatureIds(slabTuples); @@ -83,10 +99,11 @@ Result<> ComputeKernelAvgMisorientations::operator()() std::vector slabQuats(slabTuples * 4); quatsStore.copyIntoBuffer(slabStartTuple * 4, nonstd::span(slabQuats.data(), slabTuples * 4)); - // Output buffer for this plane + // Output buffer for this single Z-plane — populated locally, then + // bulk-written to the output DataStore after the plane is processed. std::vector planeOutput(sliceSize, 0.0f); - // Offset of current plane within the slab + // Index offset from the slab start to the current Z-plane within the slab const usize planeOffsetInSlab = static_cast(plane - slabZMin) * sliceSize; for(int64 row = 0; row < yPoints; row++) @@ -105,7 +122,7 @@ Result<> ComputeKernelAvgMisorientations::operator()() continue; } - // Extract center quaternion + // Extract the center voxel's quaternion from the slab buffer ebsdlib::QuatD q1; const usize q1Idx = pointInSlab * 4; q1[0] = slabQuats[q1Idx]; @@ -143,6 +160,8 @@ Result<> ComputeKernelAvgMisorientations::operator()() continue; } + // All neighbor lookups index into the slab buffers (local RAM), + // not the DataStore. This is where the OOC savings come from. const usize neighborInSlab = nzInSlab + static_cast(ny * xPoints + nx); if(slabFeatureIds[neighborInSlab] == featureId) @@ -166,7 +185,7 @@ Result<> ComputeKernelAvgMisorientations::operator()() } } - // Write this plane's output via bulk I/O + // Bulk-write this plane's computed kernel averages to the output DataStore const usize planeStartTuple = static_cast(plane) * sliceSize; outputStore.copyFromBuffer(planeStartTuple, nonstd::span(planeOutput.data(), sliceSize)); } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.hpp index 4fe32ce72d..a783a34b78 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeKernelAvgMisorientations.hpp @@ -12,19 +12,52 @@ namespace nx::core { +/** + * @brief Input values for the ComputeKernelAvgMisorientations algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeKernelAvgMisorientationsInputValues { - VectorInt32Parameter::ValueType KernelSize; - DataPath FeatureIdsArrayPath; - DataPath CellPhasesArrayPath; - DataPath QuatsArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath KernelAverageMisorientationsArrayName; - DataPath InputImageGeometry; + VectorInt32Parameter::ValueType KernelSize; ///< Half-widths {kX, kY, kZ} of the kernel in each dimension + DataPath FeatureIdsArrayPath; ///< Cell-level Int32 feature ID per voxel + DataPath CellPhasesArrayPath; ///< Cell-level Int32 phase index per voxel + DataPath QuatsArrayPath; ///< Cell-level Float32 quaternions (4 components) + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath KernelAverageMisorientationsArrayName; ///< Output: Cell-level Float32 KAM value (degrees) + DataPath InputImageGeometry; ///< ImageGeom providing voxel grid dimensions }; /** - * @class + * @class ComputeKernelAvgMisorientations + * @brief Computes the Kernel Average Misorientation (KAM) for each voxel in + * an ImageGeom. + * + * For each voxel, the misorientation angle between the voxel and every + * neighbor within the user-specified kernel is calculated (using + * crystallographic symmetry operators). The average of these angles is + * stored as the KAM value. Only neighbors belonging to the same Feature + * (same featureId) are included. + * + * ## OOC Optimization (Major Rewrite) + * + * The original implementation used `ParallelData3DAlgorithm` with per-element + * `operator[]` access, which causes catastrophic performance with OOC storage + * because the kernel neighborhood access pattern triggers random chunk + * load/evict cycles. + * + * The optimized implementation uses a **slab-based** strategy: + * - For each Z-plane, a slab spanning `[plane - kZ, plane + kZ]` is + * bulk-read via `copyIntoBuffer()`. This slab contains all data needed + * for every neighbor lookup of voxels in the current plane. + * - All kernel neighbor accesses index into local contiguous buffers + * (zero virtual dispatch overhead). + * - The output for each plane is bulk-written via `copyFromBuffer()`. + * - Ensemble-level crystal structures are cached in a local vector. + * + * The slab approach has predictable memory usage proportional to + * `(2*kZ + 1) * X * Y` and provides sequential I/O that OOC stores handle + * efficiently. Parallelization is deliberately disabled because the slab + * I/O pattern already provides good throughput and avoids thread-safety + * issues with DataStore access. */ class ORIENTATIONANALYSIS_EXPORT ComputeKernelAvgMisorientations { @@ -38,6 +71,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeKernelAvgMisorientations ComputeKernelAvgMisorientations& operator=(const ComputeKernelAvgMisorientations&) = delete; ComputeKernelAvgMisorientations& operator=(ComputeKernelAvgMisorientations&&) noexcept = delete; + /** + * @brief Executes the KAM computation using slab-based bulk I/O. + * @return Result<> with any errors encountered during execution. + */ Result<> operator()(); private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp index 492409f5fc..9f2f2217a9 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.cpp @@ -150,8 +150,11 @@ std::optional FindTwinBoundaryIncoherence(const Eigen::Vector3d& xstl_norm, c } /** - * @brief Parallel worker for twin boundaries with incoherence. - * All arrays are locally cached vectors — zero OOC virtual dispatch in the hot loop. + * @brief Parallel worker that identifies twin boundaries and computes their + * incoherence. All input arrays are passed as local std::vector references + * (pre-cached from DataStores), eliminating OOC virtual dispatch in the hot + * loop. Output is written to local uint8/float32 vectors that are later + * bulk-copied back to DataStores. */ class CalculateTwinBoundaryWithIncoherenceImpl { @@ -240,8 +243,10 @@ class CalculateTwinBoundaryWithIncoherenceImpl }; /** - * @brief Parallel worker for twin boundaries without incoherence. - * All arrays are locally cached vectors — zero OOC virtual dispatch in the hot loop. + * @brief Parallel worker that identifies twin boundaries (without computing + * incoherence). All input arrays are local std::vector references, avoiding + * OOC DataStore access during parallel execution. Output flags are written to + * a local uint8 vector. */ class CalculateTwinBoundaryImpl { @@ -328,10 +333,21 @@ const std::atomic_bool& ComputeTwinBoundaries::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Identifies twin boundaries on a triangle surface mesh by checking + * misorientation between adjacent grains against the 60-degree <111> twin + * relationship. Optionally computes the boundary incoherence angle. + * + * OOC strategy: All arrays (ensemble, feature, and face level) are bulk-read + * into local std::vectors via copyIntoBuffer before the parallel computation + * begins. The parallel workers operate entirely on these local caches with + * zero OOC virtual dispatch. After parallel execution, results are bulk-written + * back to DataStores via copyFromBuffer. + */ Result<> ComputeTwinBoundaries::operator()() { // ------------------------------------------------------------------------- - // Cache ensemble-level crystalStructures locally (tiny array) + // Bulk-read ensemble-level crystalStructures into local memory (tiny array). // ------------------------------------------------------------------------- const auto& crystalStructuresStore = m_DataStructure.getDataAs(m_InputValues->CrystalStructuresArrayPath)->getDataStoreRef(); const usize numCrystalStructures = crystalStructuresStore.getNumberOfTuples(); @@ -360,7 +376,8 @@ Result<> ComputeTwinBoundaries::operator()() } // ------------------------------------------------------------------------- - // Cache feature-level arrays locally (O(features) — thousands, not millions) + // Bulk-read feature-level arrays into local vectors (O(features)). These are + // accessed randomly by feature ID during the parallel face loop. // ------------------------------------------------------------------------- const auto& featurePhasesStore = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); const usize numFeatures = featurePhasesStore.getNumberOfTuples(); @@ -372,7 +389,9 @@ Result<> ComputeTwinBoundaries::operator()() avgQuatsStore.copyIntoBuffer(0, nonstd::span(avgQuats.data(), numFeatures * 4)); // ------------------------------------------------------------------------- - // Cache face-level arrays locally (O(faces) — scales with surface area, not volume) + // Bulk-read face-level arrays into local vectors (O(faces), scales with + // surface area rather than volume). This is the largest cache but still + // much smaller than cell-level data in a typical EBSD dataset. // ------------------------------------------------------------------------- const auto& faceLabelsStore = m_DataStructure.getDataAs(m_InputValues->FaceLabelsArrayPath)->getDataStoreRef(); const usize numFaces = faceLabelsStore.getNumberOfTuples(); @@ -389,7 +408,8 @@ Result<> ComputeTwinBoundaries::operator()() } // ------------------------------------------------------------------------- - // Output buffers — parallel workers write to these, then bulk-copy to stores + // Output buffers — parallel workers write directly into these local vectors. + // After execution completes, results are bulk-copied to the output DataStores. // ------------------------------------------------------------------------- std::vector twinBoundariesOut(numFaces, 0); std::vector twinBoundaryIncoherenceOut; @@ -402,7 +422,8 @@ Result<> ComputeTwinBoundaries::operator()() const float32 axistol = m_InputValues->AxisTolerance * Constants::k_PiF / 180.0f; // ------------------------------------------------------------------------- - // Parallel execution — all data access is on local vectors, zero OOC dispatch + // Parallel execution over all faces. The workers index only into local + // vectors, so there is zero OOC DataStore access during computation. // ------------------------------------------------------------------------- ParallelDataAlgorithm dataAlg; dataAlg.setRange(0, numFaces); @@ -419,7 +440,9 @@ Result<> ComputeTwinBoundaries::operator()() } // ------------------------------------------------------------------------- - // Write results back to DataStores via bulk I/O + // Write results from local buffers back to DataStores via bulk I/O. + // TwinBoundaries uses a MaskCompare interface (no bulk copy API), so it + // must be written element-by-element. The incoherence array uses copyFromBuffer. // ------------------------------------------------------------------------- std::unique_ptr twinBoundaries; try diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.hpp index d5ba042b09..980a7feda4 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeTwinBoundaries.hpp @@ -10,22 +10,51 @@ namespace nx::core { +/** + * @brief Input values for the ComputeTwinBoundaries algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ComputeTwinBoundariesInputValues { - bool FindCoherence; - float32 AngleTolerance; - float32 AxisTolerance; - DataPath FaceLabelsArrayPath; - DataPath FaceNormalsArrayPath; - DataPath AvgQuatsArrayPath; - DataPath FeaturePhasesArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath TwinBoundariesArrayPath; - DataPath TwinBoundaryIncoherenceArrayPath; + bool FindCoherence; ///< If true, also compute incoherence angle using face normals + float32 AngleTolerance; ///< Tolerance (degrees) for the sigma-3 misorientation angle + float32 AxisTolerance; ///< Tolerance (degrees) for the sigma-3 misorientation axis + DataPath FaceLabelsArrayPath; ///< Face-level Int32 labels (2 components: feature1, feature2) + DataPath FaceNormalsArrayPath; ///< Face-level Float64 normals (3 components, coherence mode only) + DataPath AvgQuatsArrayPath; ///< Feature-level Float32 average quaternions (4 components) + DataPath FeaturePhasesArrayPath; ///< Feature-level Int32 phase index per feature + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath TwinBoundariesArrayPath; ///< Output: Face-level mask (Bool or UInt8) flagging twin boundaries + DataPath TwinBoundaryIncoherenceArrayPath; ///< Output: Face-level Float32 incoherence angle (degrees, coherence mode) }; /** - * @class + * @class ComputeTwinBoundaries + * @brief Identifies sigma-3 twin boundaries on a triangle surface mesh and + * optionally computes the incoherence angle for each twin boundary. + * + * For each face on the surface mesh, the average orientations of the two + * adjacent Features are compared to the sigma-3 twin misorientation + * (60 degrees about <111>). If the misorientation falls within the user-specified + * axis and angle tolerances, the face is flagged as a twin boundary. When + * coherence computation is enabled, the crystal direction parallel to the face + * normal is compared with the misorientation axis to determine the incoherence. + * + * Only Cubic-High (m3m) and Cubic-Low (m3) Laue classes are considered. + * + * ## OOC Optimization + * + * All input arrays are cached entirely in local `std::vector`s before the + * parallel computation begins: + * - Feature-level arrays (phases, avgQuats) -- O(features), small. + * - Face-level arrays (faceLabels, faceNormals) -- O(faces), scales with + * surface area not volume, typically manageable. + * - Ensemble-level crystal structures -- tiny. + * + * The parallel workers (`CalculateTwinBoundaryImpl` and + * `CalculateTwinBoundaryWithIncoherenceImpl`) operate exclusively on these + * local vectors, achieving zero virtual dispatch overhead in the hot loop. + * Results are accumulated in local output vectors and written back to + * DataStores via bulk I/O after parallel execution completes. */ class ORIENTATIONANALYSIS_EXPORT ComputeTwinBoundaries { @@ -38,6 +67,10 @@ class ORIENTATIONANALYSIS_EXPORT ComputeTwinBoundaries ComputeTwinBoundaries& operator=(const ComputeTwinBoundaries&) = delete; ComputeTwinBoundaries& operator=(ComputeTwinBoundaries&&) noexcept = delete; + /** + * @brief Executes twin boundary identification with locally cached data. + * @return Result<> with any errors or warnings (e.g., NaN normals, non-cubic phases). + */ Result<> operator()(); const std::atomic_bool& getCancel(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp index c8f7d4d489..ceb41bc750 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.cpp @@ -151,6 +151,16 @@ struct StereographicCheck } }; +/** + * Macro-generated parallel convertor classes (one per target representation). + * Each class reads from an input DataStore and writes to an output DataStore, + * converting orientation representations (Euler, OM, Quat, etc.) tuple by tuple. + * + * OOC strategy: The operator() processes the assigned Range in 4096-tuple chunks. + * Each chunk bulk-reads input data via copyIntoBuffer, converts all tuples locally, + * then bulk-writes output via copyFromBuffer. This replaces per-element operator[] + * access, which would cause O(N) virtual dispatch calls into the OOC storage layer. + */ #define OC_TBB_IMPL(TO_REP) \ template \ class TO_REP##Convertor \ @@ -163,31 +173,31 @@ struct StereographicCheck } \ void operator()(const Range& r) const \ { \ - static constexpr usize k_ChunkSize = 4096; \ - const usize inNumComps = m_Input.getNumberOfComponents(); \ - const usize outNumComps = m_Output.getNumberOfComponents(); \ - const usize totalTuples = r.max() - r.min(); \ - const usize maxChunkTuples = std::min(k_ChunkSize, totalTuples); \ + static constexpr usize k_ChunkSize = 4096; \ + const usize inNumComps = m_Input.getNumberOfComponents(); \ + const usize outNumComps = m_Output.getNumberOfComponents(); \ + const usize totalTuples = r.max() - r.min(); \ + const usize maxChunkTuples = std::min(k_ChunkSize, totalTuples); \ auto inBuffer = std::make_unique(maxChunkTuples * inNumComps); \ auto outBuffer = std::make_unique(maxChunkTuples * outNumComps); \ - usize tupleIdx = r.min(); \ + usize tupleIdx = r.min(); \ while(tupleIdx < r.max()) \ { \ - const usize chunkTuples = std::min(k_ChunkSize, r.max() - tupleIdx); \ - const usize inElemCount = chunkTuples * inNumComps; \ - const usize outElemCount = chunkTuples * outNumComps; \ + const usize chunkTuples = std::min(k_ChunkSize, r.max() - tupleIdx); \ + const usize inElemCount = chunkTuples * inNumComps; \ + const usize outElemCount = chunkTuples * outNumComps; \ m_Input.copyIntoBuffer(tupleIdx* inNumComps, nonstd::span(inBuffer.get(), inElemCount)); \ InputType inputInstance; \ - for(usize t = 0; t < chunkTuples; ++t) \ + for(usize t = 0; t < chunkTuples; ++t) \ { \ - const usize inOff = t * inNumComps; \ - const usize outOff = t * outNumComps; \ - for(usize c = 0; c < inNumComps; ++c) \ + const usize inOff = t * inNumComps; \ + const usize outOff = t * outNumComps; \ + for(usize c = 0; c < inNumComps; ++c) \ { \ inputInstance[c] = inBuffer[inOff + c]; \ } \ OutputType outputInstance = inputInstance.to##TO_REP(); \ - for(usize c = 0; c < outNumComps; ++c) \ + for(usize c = 0; c < outNumComps; ++c) \ { \ outBuffer[outOff + c] = outputInstance[c]; \ } \ @@ -226,6 +236,12 @@ ConvertOrientations::ConvertOrientations(DataStructure& dataStructure, const IFi ConvertOrientations::~ConvertOrientations() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Converts an array of orientation representations from one type to + * another (e.g., Euler angles to quaternions). Supports all 8 EbsdLib + * orientation types. Parallelized via ParallelDataAlgorithm with macro- + * generated convertor classes that use chunked bulk I/O internally. + */ Result<> ConvertOrientations::operator()() { using ValidateInputDataFunctionType = std::function; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.hpp index ed0173b4db..c7240761ae 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ConvertOrientations.hpp @@ -26,19 +26,36 @@ constexpr int32 k_InputComponentCountError = -67004; constexpr int32 k_MatchingTypesError = -67005; } // namespace convert_orientations_constants +/** + * @brief Input values for the ConvertOrientations algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ConvertOrientationsInputValues { - ArraySelectionParameter::ValueType InputOrientationArrayPath; - ebsdlib::orientations::Type InputType; - DataObjectNameParameter::ValueType OutputOrientationArrayName; - ebsdlib::orientations::Type OutputType; + ArraySelectionParameter::ValueType InputOrientationArrayPath; ///< Cell-level Float32 input orientation array + ebsdlib::orientations::Type InputType; ///< Enumerated input representation type + DataObjectNameParameter::ValueType OutputOrientationArrayName; ///< Name for the output orientation array + ebsdlib::orientations::Type OutputType; ///< Enumerated output representation type }; /** * @class ConvertOrientations - * @brief This algorithm implements support code for the ConvertOrientationsFilter + * @brief Converts between orientation representations (Euler angles, quaternions, + * orientation matrices, axis-angle, Rodrigues, homochoric, cubochoric, + * and stereographic projection). + * + * A macro-generated parallel worker class is instantiated for each valid + * input/output combination. The worker reads input tuples, converts each + * orientation, and writes the result to the output array. + * + * ## OOC Optimization + * + * The macro-generated parallel worker classes now use chunked bulk I/O + * internally (chunk size of 4096 tuples). Within each `operator()(Range)` + * call, input data is read via `copyIntoBuffer()` and output data is written + * via `copyFromBuffer()` in chunks, with the conversion loop operating on + * contiguous local buffers. This replaces per-element `operator[]` access + * that would trigger chunk load/evict cycles with OOC storage. */ - class ORIENTATIONANALYSIS_EXPORT ConvertOrientations { public: @@ -50,6 +67,10 @@ class ORIENTATIONANALYSIS_EXPORT ConvertOrientations ConvertOrientations& operator=(const ConvertOrientations&) = delete; ConvertOrientations& operator=(ConvertOrientations&&) noexcept = delete; + /** + * @brief Executes the orientation conversion using parallel chunked bulk I/O. + * @return Result<> with any errors encountered during execution. + */ Result<> operator()(); private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp index 8409e3ba5e..c3f10b8f99 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/EBSDSegmentFeatures.hpp @@ -18,88 +18,191 @@ namespace nx::core { /** - * @brief The EBSDSegmentFeaturesInputValues struct + * @struct EBSDSegmentFeaturesInputValues + * @brief Holds all user-supplied parameters for the EBSDSegmentFeatures algorithm. */ struct ORIENTATIONANALYSIS_EXPORT EBSDSegmentFeaturesInputValues { - float32 MisorientationTolerance; - bool UseMask; - bool RandomizeFeatureIds; - SegmentFeatures::NeighborScheme NeighborScheme; - DataPath ImageGeometryPath; - DataPath QuatsArrayPath; - DataPath CellPhasesArrayPath; - DataPath MaskArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath FeatureIdsArrayPath; - DataPath CellFeatureAttributeMatrixPath; - DataPath ActiveArrayPath; - bool IsPeriodic; + float32 MisorientationTolerance = 0.0f; ///< Maximum misorientation angle (radians) for grouping voxels into the same feature + bool UseMask = false; ///< Whether to exclude masked voxels from segmentation + bool RandomizeFeatureIds = false; ///< Whether to randomize feature IDs after segmentation for better visual contrast + SegmentFeatures::NeighborScheme NeighborScheme{}; ///< Face-only (6) or all-connected (26) neighbor connectivity + DataPath ImageGeometryPath; ///< Path to the IGridGeometry defining the 3D voxel grid + DataPath QuatsArrayPath; ///< Path to the Float32 quaternion array (4 components per cell) + DataPath CellPhasesArrayPath; ///< Path to the Int32 cell phases array + DataPath MaskArrayPath; ///< Path to the Bool/UInt8 mask array (only used when UseMask is true) + DataPath CrystalStructuresArrayPath; ///< Path to the UInt32 crystal structures ensemble array + DataPath FeatureIdsArrayPath; ///< Path to the output Int32 feature IDs array + DataPath CellFeatureAttributeMatrixPath; ///< Path to the Feature-level AttributeMatrix (resized to match feature count) + DataPath ActiveArrayPath; ///< Path to the output UInt8 Active array (1 = active feature, 0 = reserved slot 0) + bool IsPeriodic = false; ///< Whether to apply periodic boundary conditions during segmentation }; /** - * @brief + * @class EBSDSegmentFeatures + * @brief Segments an EBSD dataset into crystallographic features (grains) + * by flood-filling contiguous voxels whose orientations are within a + * user-specified misorientation tolerance. + * + * Two neighboring voxels are grouped into the same feature only if they share + * the same crystallographic phase and their misorientation -- computed via the + * appropriate EbsdLib LaueOps symmetry operator from their quaternion + * representations -- is below MisorientationTolerance. The misorientation + * calculation accounts for all symmetry-equivalent orientations of the given + * Laue class, so that the minimum rotation angle between the two crystal + * orientations is used. + * + * ## Algorithm Dispatch + * + * The operator() dispatches between two segmentation strategies: + * - **In-core (DFS flood fill)**: The classic depth-first-search approach + * inherited from SegmentFeatures::execute(). Accesses DataArrays directly + * via operator[] for each voxel and neighbor. + * - **Out-of-core (CCL)**: A connected-component labeling algorithm via + * SegmentFeatures::executeCCL() that processes data slice-by-slice. + * Selected when IsOutOfCore() detects the FeatureIds array is backed by + * an on-disk DataStore, or when ForceOocAlgorithm() is set for testing. + * + * ## Rolling 2-Slot Slice Buffers (OOC Optimization) + * + * The CCL algorithm calls isValidVoxel() and areNeighborsSimilar() for every + * voxel and its neighbors. On OOC DataStores, each call would trigger a chunk + * decompress-read-evict cycle, making the algorithm orders of magnitude slower. + * + * To avoid this, prepareForSlice() bulk-reads each Z-slice's quaternion, phase, + * and mask data into a rolling 2-slot buffer (even slices -> slot 0, odd slices + * -> slot 1). Because CCL processes slices sequentially and only compares + * adjacent slices, both the current slice (iz) and previous slice (iz-1) are + * always resident. The isValidVoxel() and areNeighborsSimilar() methods check + * the buffer first (fast path) and fall back to direct DataStore access only + * when the needed slice is not buffered (e.g., during periodic boundary merging). + * + * Additionally, the small ensemble-level crystal structures array is cached + * locally in m_CrystalStructuresCache to avoid per-voxel virtual dispatch + * through the DataStore. */ class ORIENTATIONANALYSIS_EXPORT EBSDSegmentFeatures : public SegmentFeatures { public: - using FeatureIdsArrayType = Int32Array; + using FeatureIdsArrayType = Int32Array; ///< Type alias for the feature IDs array + /** + * @brief Constructs the algorithm with all required references and parameters. + * @param dataStructure The DataStructure containing all input/output arrays. + * @param mesgHandler Handler for sending progress messages to the UI. + * @param shouldCancel Atomic flag checked between iterations to support cancellation. + * @param inputValues User-supplied parameters controlling the segmentation behavior. + */ EBSDSegmentFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, EBSDSegmentFeaturesInputValues* inputValues); ~EBSDSegmentFeatures() noexcept override; - EBSDSegmentFeatures(const EBSDSegmentFeatures&) = delete; // Copy Constructor Not Implemented - EBSDSegmentFeatures(EBSDSegmentFeatures&&) = delete; // Move Constructor Not Implemented - EBSDSegmentFeatures& operator=(const EBSDSegmentFeatures&) = delete; // Copy Assignment Not Implemented - EBSDSegmentFeatures& operator=(EBSDSegmentFeatures&&) = delete; // Move Assignment Not Implemented + EBSDSegmentFeatures(const EBSDSegmentFeatures&) = delete; + EBSDSegmentFeatures(EBSDSegmentFeatures&&) = delete; + EBSDSegmentFeatures& operator=(const EBSDSegmentFeatures&) = delete; + EBSDSegmentFeatures& operator=(EBSDSegmentFeatures&&) = delete; + /** + * @brief Executes the EBSD segmentation algorithm, dispatching between DFS + * (in-core) and CCL (OOC) strategies based on the backing DataStore type. + * @return Result<> indicating success or any errors encountered during execution. + */ Result<> operator()(); protected: + /** + * @brief Finds the next unassigned voxel to serve as a seed for a new feature. + * Used by the DFS flood-fill path (execute()). + * @param gnum The feature number to assign to the seed voxel. + * @param nextSeed Linear index to start scanning from. + * @return Linear index of the seed voxel, or -1 if no more seeds exist. + */ int64 getSeed(int32 gnum, int64 nextSeed) const override; + + /** + * @brief Determines whether a neighbor should be merged into the current + * feature during DFS flood fill. Checks mask, phase equality, and + * crystallographic misorientation via LaueOps. + * @param referencePoint Linear index of the current (reference) voxel. + * @param neighborPoint Linear index of the candidate neighbor voxel. + * @param gnum The feature number to assign if the neighbor is accepted. + * @return true if the neighbor was merged (its featureId set to gnum). + */ bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; /** - * @brief Checks whether a voxel can participate in EBSD segmentation based on mask and phase. + * @brief Checks whether a voxel can participate in EBSD segmentation. + * + * Used by the CCL path. When slice buffers are active, reads from the + * in-memory buffer (fast path); otherwise falls back to direct DataStore + * access (OOC fallback). + * * @param point Linear voxel index. - * @return true if the voxel passes mask and phase checks. + * @return true if the voxel passes mask and phase > 0 checks. */ bool isValidVoxel(int64 point) const override; /** - * @brief Determines whether two neighboring voxels belong to the same EBSD segment. + * @brief Determines whether two neighboring voxels belong to the same EBSD + * segment, used by the CCL path. + * + * When both voxels' Z-slices are in the rolling buffer, all data is read + * from local memory. Otherwise falls back to direct DataStore access. + * * @param point1 First voxel index. * @param point2 Second (neighbor) voxel index. - * @return true if both voxels share the same phase and their misorientation is within tolerance. + * @return true if both share the same phase and their misorientation is within tolerance. */ bool areNeighborsSimilar(int64 point1, int64 point2) const override; + /** + * @brief Pre-loads a Z-slice's quaternion, phase, and mask data into the + * rolling 2-slot buffer for OOC-efficient access during CCL. + * + * Slot assignment: even slices -> slot 0, odd slices -> slot 1. Passing + * iz < 0 disables slice buffering (used after the slice-by-slice sweep + * completes, before periodic boundary merging). + * + * @param iz Z-slice index to load, or -1 to disable buffering. + * @param dimX Number of voxels in X. + * @param dimY Number of voxels in Y. + * @param dimZ Number of voxels in Z. + */ void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) override; private: - const EBSDSegmentFeaturesInputValues* m_InputValues = nullptr; - Float32Array* m_QuatsArray = nullptr; - FeatureIdsArrayType* m_CellPhases = nullptr; - std::unique_ptr m_GoodVoxelsArray = nullptr; - DataArray* m_CrystalStructures = nullptr; + const EBSDSegmentFeaturesInputValues* m_InputValues = nullptr; ///< Non-owning pointer to user-supplied parameters + Float32Array* m_QuatsArray = nullptr; ///< Pointer to the quaternion array (4 components per cell) + FeatureIdsArrayType* m_CellPhases = nullptr; ///< Pointer to the cell phases array + std::unique_ptr m_GoodVoxelsArray = nullptr; ///< Mask comparator for filtering valid voxels + DataArray* m_CrystalStructures = nullptr; ///< Pointer to the crystal structures ensemble array - FeatureIdsArrayType* m_FeatureIdsArray = nullptr; + FeatureIdsArrayType* m_FeatureIdsArray = nullptr; ///< Pointer to the output feature IDs array - std::vector m_OrientationOps; + std::vector m_OrientationOps; ///< Cached Laue symmetry operators for all crystal systems + /** + * @brief Allocates the rolling 2-slot slice buffers for OOC optimization. + * Each slot holds one full XY slice of quaternions, phases, and mask flags. + * @param dimX Number of voxels in X. + * @param dimY Number of voxels in Y. + */ void allocateSliceBuffers(int64 dimX, int64 dimY); + + /** + * @brief Releases the slice buffers and resets m_UseSliceBuffers to false. + */ void deallocateSliceBuffers(); // Rolling 2-slot input buffers for OOC optimization. // Pre-loading input data into these avoids per-element OOC overhead // during neighbor comparisons in the CCL algorithm. - std::vector m_QuatBuffer; - std::vector m_PhaseBuffer; - std::vector m_MaskBuffer; - std::vector m_CrystalStructuresCache; - int64 m_BufSliceSize = 0; - int64 m_BufferedSliceZ[2] = {-1, -1}; - bool m_UseSliceBuffers = false; + std::vector m_QuatBuffer; ///< 2 * sliceSize * 4 quaternion components + std::vector m_PhaseBuffer; ///< 2 * sliceSize phase IDs + std::vector m_MaskBuffer; ///< 2 * sliceSize mask flags + std::vector m_CrystalStructuresCache; ///< Local copy of the crystal structures ensemble array + int64 m_BufSliceSize = 0; ///< Number of voxels per XY slice (dimX * dimY) + int64 m_BufferedSliceZ[2] = {-1, -1}; ///< Z-indices currently loaded in each buffer slot + bool m_UseSliceBuffers = false; ///< Whether slice buffers are active }; } // namespace nx::core diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp index e82c5a5a46..f9c09836cf 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.cpp @@ -172,6 +172,19 @@ void MergeTwins::groupFeaturesExecute() } // ----------------------------------------------------------------------------- +/** + * @brief Merges twin-related features by clustering grains that share a 60-degree + * <111> misorientation (sigma-3 twin relationship). The algorithm first identifies + * twin pairs using the inherited groupFeatures framework, then assigns parent IDs + * to every voxel based on the feature-to-parent mapping. + * + * OOC strategy: The cellParentIds array is initialized via chunked copyFromBuffer + * (rather than a single fill() call) to avoid per-element OOC overhead. The + * feature-to-parent mapping is cached in a local vector, then the voxel-level + * parent assignment loop reads featureIds in 64K-tuple chunks via copyIntoBuffer, + * looks up parents from the local cache, and bulk-writes cellParentIds via + * copyFromBuffer. + */ Result<> MergeTwins::operator()() { Result result = {}; @@ -189,7 +202,9 @@ Result<> MergeTwins::operator()() usize totalPoints = cellParentIdsStore.getNumberOfTuples(); - // Chunked fill of cellParentIds for OOC efficiency + // Initialize cellParentIds to -1 using chunked bulk writes. For OOC stores, + // a single fill() call would trigger per-element virtual dispatch; chunked + // copyFromBuffer amortizes the overhead over 64K-tuple writes. { constexpr usize k_FillChunk = 65536; std::vector fillBuf(k_FillChunk, -1); @@ -230,12 +245,16 @@ Result<> MergeTwins::operator()() result, ConvertResult(MakeErrorResult(-23501, "The number of grouped Features was 0 or 1 which means no grouped Features were detected. A grouping value may be set too high"))); } - // Cache feature-level featureParentIds locally for chunked voxel loop + // Cache feature-level featureParentIds into a local vector. The voxel loop + // below looks up the parent for each voxel's feature (random access by + // featureId), which would thrash OOC chunks if done via DataStore operator[]. const usize numFeatures = featureParentIds.getNumberOfTuples(); std::vector featureParentIdsCache(numFeatures); featureParentIds.copyIntoBuffer(0, nonstd::span(featureParentIdsCache.data(), numFeatures)); - // Chunked bulk I/O for voxel-level parent ID assignment + // Assign parent IDs to every voxel using chunked bulk I/O: read a chunk of + // featureIds, look up each feature's parent from the local cache, then + // bulk-write the parent IDs back to the cellParentIds DataStore. int32 numParents = 0; { constexpr usize k_ChunkSize = 65536; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.hpp index b9d8286497..18752bd463 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/MergeTwins.hpp @@ -12,25 +12,46 @@ namespace nx::core { +/** + * @brief Input values for the MergeTwins algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT MergeTwinsInputValues { - DataPath ContiguousNeighborListArrayPath; - float32 AxisTolerance; - float32 AngleTolerance; - DataPath FeaturePhasesArrayPath; - DataPath AvgQuatsArrayPath; - DataPath FeatureIdsArrayPath; - DataPath CrystalStructuresArrayPath; - DataPath CellParentIdsArrayPath; - DataPath NewCellFeatureAttributeMatrixPath; - DataPath FeatureParentIdsArrayPath; - DataPath ActiveArrayPath; - uint64 Seed; - bool RandomizeParentIds = false; + DataPath ContiguousNeighborListArrayPath; ///< Feature-level NeighborList of contiguous neighbors + float32 AxisTolerance; ///< Tolerance (degrees) for the twin axis direction + float32 AngleTolerance; ///< Tolerance (degrees) for the twin misorientation angle + DataPath FeaturePhasesArrayPath; ///< Feature-level Int32 phase index per feature + DataPath AvgQuatsArrayPath; ///< Feature-level Float32 average quaternions (4 components) + DataPath FeatureIdsArrayPath; ///< Cell-level Int32 feature ID per voxel + DataPath CrystalStructuresArrayPath; ///< Ensemble-level UInt32 crystal structure Laue classes + DataPath CellParentIdsArrayPath; ///< Output: Cell-level Int32 parent feature ID + DataPath NewCellFeatureAttributeMatrixPath; ///< Output: AttributeMatrix for new parent features + DataPath FeatureParentIdsArrayPath; ///< Output: Feature-level Int32 parent feature ID + DataPath ActiveArrayPath; ///< Output: Feature-level bool active status + uint64 Seed; ///< Random seed for parent ID randomization + bool RandomizeParentIds = false; ///< Whether to randomize the order of parent IDs }; /** - * @class + * @class MergeTwins + * @brief Groups neighboring Features that share a sigma-3 twin relationship + * (FCC, 60 degrees about <111>) into parent features. + * + * Only Cubic-High (m3m) Laue class features are considered. The algorithm + * compares average orientations of neighboring features against the sigma-3 + * twin misorientation within user-specified axis and angle tolerances. + * + * ## OOC Optimization + * + * The voxel-level pass that assigns cellParentIds from featureParentIds is + * the only cell-level operation. It uses chunked bulk I/O: + * - `cellParentIds` is filled with -1 in chunks via `copyFromBuffer()`. + * - `featureIds` are read in chunks of 65536 via `copyIntoBuffer()`. + * - `featureParentIds` are cached locally (feature-level, small). + * - The computed `cellParentIds` are written back in matching chunks. + * + * This avoids per-voxel virtual dispatch that would cause OOC chunk thrashing + * during the cell-level parent ID assignment loop. */ class ORIENTATIONANALYSIS_EXPORT MergeTwins { @@ -43,6 +64,10 @@ class ORIENTATIONANALYSIS_EXPORT MergeTwins MergeTwins& operator=(const MergeTwins&) = delete; MergeTwins& operator=(MergeTwins&&) noexcept = delete; + /** + * @brief Executes twin merging and assigns parent IDs using chunked bulk I/O. + * @return Result<> with any errors or warnings encountered. + */ Result<> operator()(); const std::atomic_bool& getCancel(); @@ -58,8 +83,11 @@ class ORIENTATIONANALYSIS_EXPORT MergeTwins std::mt19937_64 m_Generator = {}; std::uniform_real_distribution m_Distribution = {}; + /** @brief Iterates over features, grouping twins into parent features. */ void groupFeaturesExecute(); + /** @brief Returns the seed feature for a new parent group. */ int getSeed(int32 newFid); + /** @brief Tests if two features satisfy the sigma-3 twin relationship. */ bool determineGrouping(int32 referenceFeature, int32 neighborFeature, int32 newFid); }; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp index 5541992edb..95a29b838f 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.cpp @@ -15,6 +15,9 @@ using namespace nx::core; using FloatVec3Type = std::vector; // ----------------------------------------------------------------------------- +/** + * @brief Constructs ReadAngData with the given DataStructure, message handler, cancel flag, and input values. + */ ReadAngData::ReadAngData(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ReadAngDataInputValues* inputValues) : m_DataStructure(dataStructure) , m_MessageHandler(msgHandler) @@ -27,6 +30,14 @@ ReadAngData::ReadAngData(DataStructure& dataStructure, const IFilter::MessageHan ReadAngData::~ReadAngData() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Reads a .ang EBSD data file and populates the DataStructure with the parsed arrays. + * + * Delegates to EbsdLib's AngReader for file parsing, then calls loadMaterialInfo() + * to populate ensemble-level arrays and copyRawEbsdData() to populate cell-level arrays. + * + * @return Result<> indicating success or an error from the EbsdLib reader. + */ Result<> ReadAngData::operator()() { ebsdlib::AngReader reader; @@ -101,6 +112,27 @@ std::pair ReadAngData::loadMaterialInfo(ebsdlib::AngReader* } // ----------------------------------------------------------------------------- +/** + * @brief Copies raw EBSD data from the EbsdLib AngReader buffers into the DataStructure arrays. + * + * @section ooc_strategy OOC Strategy + * The EbsdLib reader holds the parsed data in contiguous in-memory buffers. We need to + * transfer this data into DataStore-backed arrays that may be out-of-core. Rather than + * using per-element operator[] (which would trigger a chunk load/evict per write on OOC + * stores), we use copyFromBuffer() to write entire contiguous ranges in single bulk + * operations. + * + * For single-component arrays (ImageQuality, ConfidenceIndex, etc.), a single + * copyFromBuffer() call writes all values at once since the reader buffer is already + * contiguous. + * + * For the Euler angles (3 separate source arrays that must be interleaved into a + * 3-component destination), we use a chunked approach: interleave k_ChunkSize tuples + * into a local buffer, then copyFromBuffer() the chunk. This bounds memory usage to + * ~768 KB (65536 * 3 * sizeof(float32)) while still achieving bulk I/O efficiency. + * + * @param reader Pointer to the EbsdLib AngReader that has already parsed the file. + */ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const { const DataPath CellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName); @@ -111,7 +143,8 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const // Prepare the Cell Attribute Matrix with the correct number of tuples based on the total Cells being read from the file. std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; - // Adjust the values of the 'phase' data to correct for invalid values and assign the read Phase Data into the actual DataArray + // Adjust the values of the 'phase' data to correct for invalid values, then bulk-write + // via copyFromBuffer (OOC-safe: single I/O call for the entire array) { if(m_ShouldCancel) { @@ -119,7 +152,7 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const } auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::Phases)); auto* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::PhaseData)); - // Validate phases in-place (original code also modifies phasePtr) + // Validate phases in-place in the reader's buffer before bulk-writing for(usize i = 0; i < totalCells; i++) { if(phasePtr[i] < 1) @@ -127,10 +160,13 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const phasePtr[i] = 1; } } + // OOC-safe: single bulk write of the entire phase array targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(phasePtr, totalCells)); } - // Condense the Euler Angles from 3 separate arrays into a single 1x3 array + // Condense the Euler Angles from 3 separate source arrays (Phi1, Phi, Phi2) into a + // single interleaved 3-component destination array. Uses chunked interleaving to + // bound memory while still writing bulk chunks via copyFromBuffer. { if(m_ShouldCancel) { @@ -143,6 +179,9 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const auto& cellEulerAngles = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::AngFile::EulerAngles)); auto& eulerStore = cellEulerAngles.getDataStoreRef(); + // Chunked interleaving: interleave k_ChunkSize tuples into a local buffer, + // then bulk-write the chunk. This avoids both per-element OOC access and + // allocating a buffer for the entire volume. constexpr usize k_ChunkSize = 65536; std::vector eulerBuf(k_ChunkSize * 3); for(usize offset = 0; offset < totalCells; offset += k_ChunkSize) @@ -163,6 +202,9 @@ void ReadAngData::copyRawEbsdData(ebsdlib::AngReader* reader) const return; } + // OOC-safe bulk writes for single-component float arrays. + // Each copyFromBuffer() writes the entire reader buffer in one I/O operation, + // which is optimal for both in-memory and OOC DataStore backends. { auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ang::ImageQuality)); auto& targetArray = m_DataStructure.getDataRefAs(CellAttributeMatrixPath.createChildPath(ebsdlib::Ang::ImageQuality)); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp index c49f43e06e..665214ef2d 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadAngData.hpp @@ -13,12 +13,15 @@ namespace nx::core { +/** + * @brief Input values for the ReadAngData algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT ReadAngDataInputValues { - std::filesystem::path InputFile; - DataPath DataContainerName; - std::string CellAttributeMatrixName; - std::string CellEnsembleAttributeMatrixName; + std::filesystem::path InputFile; ///< Path to the .ang EBSD data file. + DataPath DataContainerName; ///< Path to the output DataContainer (ImageGeom). + std::string CellAttributeMatrixName; ///< Name of the cell-level AttributeMatrix. + std::string CellEnsembleAttributeMatrixName; ///< Name of the ensemble-level AttributeMatrix. }; struct ORIENTATIONANALYSIS_EXPORT Ang_Private_Data @@ -52,8 +55,16 @@ class ORIENTATIONANALYSIS_EXPORT ReadAngDataPrivate /** * @class ReadAngData - * @brief This filter will read a single .ang file into a new Image Geometry, allowing the immediate use of Filters on the data instead of having to generate the intermediate - * .h5ebsd file. + * @brief Algorithm that reads a single .ang EBSD file into an Image Geometry. + * + * Parses the .ang file using EbsdLib's AngReader, then transfers the parsed data + * into the DataStructure's cell-level and ensemble-level arrays. + * + * @section ooc_summary OOC Optimization Summary + * All data transfer from the EbsdLib reader buffers into the DataStructure uses + * copyFromBuffer() bulk writes instead of per-element operator[] access. Euler angles + * (3 separate source arrays interleaved into 1 destination) use a chunked buffer approach + * to bound memory while maintaining bulk I/O efficiency. See copyRawEbsdData() for details. */ class ORIENTATIONANALYSIS_EXPORT ReadAngData { @@ -61,11 +72,15 @@ class ORIENTATIONANALYSIS_EXPORT ReadAngData ReadAngData(DataStructure& dataStructure, const IFilter::MessageHandler& msgHandler, const std::atomic_bool& shouldCancel, ReadAngDataInputValues* inputValues); ~ReadAngData() noexcept; - ReadAngData(const ReadAngData&) = delete; // Copy Constructor Not Implemented - ReadAngData(ReadAngData&&) = delete; // Move Constructor Not Implemented - ReadAngData& operator=(const ReadAngData&) = delete; // Copy Assignment Not Implemented - ReadAngData& operator=(ReadAngData&&) = delete; // Move Assignment Not Implemented + ReadAngData(const ReadAngData&) = delete; + ReadAngData(ReadAngData&&) = delete; + ReadAngData& operator=(const ReadAngData&) = delete; + ReadAngData& operator=(ReadAngData&&) = delete; + /** + * @brief Executes the algorithm: reads the .ang file and populates the DataStructure. + * @return Result<> indicating success or an EbsdLib error. + */ Result<> operator()(); private: @@ -75,15 +90,15 @@ class ORIENTATIONANALYSIS_EXPORT ReadAngData const ReadAngDataInputValues* m_InputValues = nullptr; /** - * @brief - * @param reader - * @return Error code. + * @brief Loads phase/crystal structure information from the reader into ensemble-level arrays. + * @param reader The EbsdLib AngReader that has already parsed the file. + * @return Pair of (error code, error message). Error code 0 indicates success. */ std::pair loadMaterialInfo(ebsdlib::AngReader* reader) const; /** - * @brief - * @param reader + * @brief Transfers raw EBSD data from reader buffers to DataStructure arrays using OOC-safe bulk I/O. + * @param reader The EbsdLib AngReader that has already parsed the file. */ void copyRawEbsdData(ebsdlib::AngReader* reader) const; }; diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp index 244bccaa95..ea3c1a5e0c 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.cpp @@ -99,6 +99,20 @@ std::pair ReadCtfData::loadMaterialInfo(ebsdlib::CtfReader* } // ----------------------------------------------------------------------------- +/** + * @brief Copies raw EBSD data from the EbsdLib CtfReader buffers into the DataStructure arrays. + * + * @section ooc_strategy OOC Strategy + * Same bulk I/O approach as ReadAngData::copyRawEbsdData(): + * - Single-component arrays use one copyFromBuffer() call each. + * - Euler angles use chunked interleaving with hex correction and optional degree-to-radian + * conversion applied in-buffer before each chunk write. + * - The crystal structures array is cached locally via copyIntoBuffer() because it is + * ensemble-level (tiny) and is needed for every cell during hex correction checks. + * Reading it once avoids repeated OOC lookups during the per-cell loop. + * + * @param reader Pointer to the EbsdLib CtfReader that has already parsed the file. + */ void ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const { const DataPath cellAttributeMatrixPath = m_InputValues->DataContainerName.createChildPath(m_InputValues->CellAttributeMatrixName); @@ -110,19 +124,22 @@ void ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const // Prepare the Cell Attribute Matrix with the correct number of tuples based on the total Cells being read from the file. std::vector tDims = {imageGeom.getNumXCells(), imageGeom.getNumYCells(), imageGeom.getNumZCells()}; - // Copy the Phase Array + // OOC-safe bulk write of the Phase array (single copyFromBuffer call) { auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::Phases)); auto* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Phase)); targetArray.getDataStoreRef().copyFromBuffer(0, nonstd::span(phasePtr, totalCells)); } - // Condense the Euler Angles from 3 separate arrays into a single 1x3 array + // Condense 3 separate Euler angle arrays into a single interleaved 3-component array, + // applying hex correction and degree-to-radian conversion as needed. { auto& crystalStructures = m_DataStructure.getDataRefAs(cellEnsembleAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::CrystalStructures)); const auto* phasePtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Phase)); - // Cache ensemble-level crystal structures locally + // Cache ensemble-level crystal structures locally via bulk read. + // This array is tiny (one entry per phase) but is accessed for every cell + // during hex correction -- caching avoids repeated OOC lookups. const auto& csStore = crystalStructures.getDataStoreRef(); const usize numPhases = csStore.getNumberOfTuples(); std::vector csCache(numPhases); @@ -135,6 +152,7 @@ void ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const auto& cellEulerAngles = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::CtfFile::EulerAngles)); auto& eulerStore = cellEulerAngles.getDataStoreRef(); + // Chunked interleaving with corrections applied in the local buffer before bulk write constexpr usize k_ChunkSize = 65536; std::vector eulerBuf(k_ChunkSize * 3); for(usize offset = 0; offset < totalCells; offset += k_ChunkSize) @@ -164,6 +182,8 @@ void ReadCtfData::copyRawEbsdData(ebsdlib::CtfReader* reader) const } } + // OOC-safe bulk writes for remaining single-component arrays. + // Each copyFromBuffer() call writes the entire reader buffer in one I/O operation. { auto* srcPtr = reinterpret_cast(reader->getPointerByName(ebsdlib::Ctf::Bands)); auto& targetArray = m_DataStructure.getDataRefAs(cellAttributeMatrixPath.createChildPath(ebsdlib::Ctf::Bands)); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp index dba1e637d5..b4d58cde2a 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadCtfData.hpp @@ -54,10 +54,16 @@ class ORIENTATIONANALYSIS_EXPORT ReadCtfDataPrivate /** * @class ReadCtfData - * @brief This filter will read a single .ctf file into a new Image Geometry, allowing the immediate use of Filters on the data instead of having to generate the - * intermediate .h5ebsd file. + * @brief Algorithm that reads a single .ctf (Oxford/HKL) EBSD file into an Image Geometry. + * + * Parses the .ctf file using EbsdLib's CtfReader, then transfers the parsed data + * into the DataStructure's cell-level and ensemble-level arrays. + * + * @section ooc_summary OOC Optimization Summary + * All data transfer uses copyFromBuffer() bulk writes. Euler angles are interleaved from + * 3 source arrays using chunked buffers with optional hex correction and degree-to-radian + * conversion. Crystal structures are cached locally for the per-cell hex correction check. */ - class ORIENTATIONANALYSIS_EXPORT ReadCtfData { public: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp index f1d6a76057..12fbe66df2 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.cpp @@ -108,6 +108,22 @@ nx::core::Result<> LoadInfo(const nx::core::ReadH5EbsdInputValues* mInputValues, return {}; } +/** + * @brief Copies selected EBSD data arrays from the H5Ebsd reader into the DataStructure. + * + * For each selected array name, retrieves the reader's in-memory buffer and bulk-writes it + * into the corresponding DataArray via copyFromBuffer(). This is OOC-safe: one bulk I/O + * operation per array regardless of the underlying DataStore backend. + * + * @tparam H5EbsdReaderType The H5Ebsd volume reader type (H5AngVolumeReader or H5CtfVolumeReader). + * @tparam T The element type of the arrays to copy (float32 or int). + * @param dataStructure The DataStructure containing the destination arrays. + * @param ebsdReader The H5Ebsd reader holding parsed data buffers. + * @param arrayNames All possible array names of this type. + * @param selectedArrayNames Set of array names the user selected for import. + * @param cellAttributeMatrixPath Parent path for the cell-level arrays. + * @param totalPoints Total number of voxels in the volume. + */ template void CopyData(nx::core::DataStructure& dataStructure, H5EbsdReaderType* ebsdReader, const std::vector& arrayNames, std::set selectedArrayNames, const nx::core::DataPath& cellAttributeMatrixPath, usize totalPoints) @@ -120,6 +136,7 @@ void CopyData(nx::core::DataStructure& dataStructure, H5EbsdReaderType* ebsdRead T* source = reinterpret_cast(ebsdReader->getPointerByName(arrayName)); nx::core::DataPath dataPath = cellAttributeMatrixPath.createChildPath(arrayName); auto& destination = dataStructure.getDataRefAs(dataPath); + // OOC-safe: single bulk write of the entire array destination.getDataStoreRef().copyFromBuffer(0, nonstd::span(source, totalPoints * destination.getNumberOfComponents())); } } @@ -218,12 +235,18 @@ nx::core::Result<> LoadEbsdData(const nx::core::ReadH5EbsdInputValues* mInputVal { degToRad = nx::core::numbers::pi_v / 180.0F; } - // Interleave 3 separate Euler arrays into [e0,e1,e2, e0,e1,e2, ...] layout using a chunked buffer - // Also apply Oxford hex correction if needed (avoids a second pass over the data) + // Interleave 3 separate Euler arrays into [e0,e1,e2, e0,e1,e2, ...] layout using a chunked buffer. + // Also apply Oxford hex correction if needed (avoids a second pass over the data). + // + // OOC note: The chunked approach writes k_ChunkTuples interleaved tuples per copyFromBuffer() + // call, bounding memory to ~768 KB while maintaining bulk I/O efficiency. constexpr usize k_ChunkTuples = 65536; auto eulerBuf = std::make_unique(k_ChunkTuples * 3); - // Cache phase data and crystal structures locally if Oxford hex correction is needed + // Cache phase data and crystal structures locally if Oxford hex correction is needed. + // The phase array (cell-level, potentially large) is cached via copyIntoBuffer() to + // avoid per-element OOC lookups during the correction loop. The crystal structures + // array (ensemble-level, tiny) is also cached for the same reason. std::unique_ptr phaseCache; std::unique_ptr xtalCache; bool applyHexCorrection = (manufacturer == ebsdlib::Ctf::Manufacturer && phaseDataArrayPtr != nullptr); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.hpp index f4008c79ea..d9b1ef3ccb 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5Ebsd.hpp @@ -44,7 +44,17 @@ struct ORIENTATIONANALYSIS_EXPORT ReadH5EbsdInputValues }; /** - * @brief The ReadH5Ebsd class + * @class ReadH5Ebsd + * @brief Algorithm that reads H5Ebsd-format EBSD data (TSL .ang or Oxford .ctf stored in HDF5). + * + * Supports both TSL (H5AngVolumeReader) and Oxford (H5CtfVolumeReader) manufacturers. + * After data import, optionally applies recommended sample/Euler reference frame rotations. + * + * @section ooc_summary OOC Optimization Summary + * The CopyData helper uses copyFromBuffer() for each selected array (single bulk write). + * Euler angle interleaving uses chunked buffers with optional hex correction and + * degree-to-radian conversion. Phase and crystal structure arrays are cached locally + * via copyIntoBuffer() when needed for per-cell correction lookups. */ class ReadH5Ebsd { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp index 50ebf8af0c..87a044c652 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.cpp @@ -24,6 +24,24 @@ Result<> ReadH5EspritData::operator()() } // ----------------------------------------------------------------------------- +/** + * @brief Copies raw EBSD data from the H5Esprit reader buffers into the DataStructure arrays. + * + * This is called once per slice (z-layer) in a multi-slice H5OINA/Esprit dataset. + * The offset parameter positions each slice's data at the correct location in the + * volume-wide arrays. + * + * @section ooc_strategy OOC Strategy + * Same approach as ReadAngData and ReadCtfData: + * - Euler angles: chunked interleaving of 3 source arrays into a 3-component destination + * with optional degree-to-radian conversion, written via copyFromBuffer() per chunk. + * - Single-component arrays (MAD, Phase, etc.): one copyFromBuffer() call each, writing + * the entire per-slice reader buffer in a single bulk operation. The offset parameter + * ensures each slice lands at the correct position in the volume-wide array. + * + * @param index The zero-based slice index, used to compute the tuple offset for this slice. + * @return Result<> indicating success or an error if pattern data is missing. + */ Result<> ReadH5EspritData::copyRawEbsdData(int index) { const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); @@ -58,7 +76,9 @@ Result<> ReadH5EspritData::copyRawEbsdData(int index) const auto* yBm = reinterpret_cast(m_Reader->getPointerByName(ebsdlib::H5Esprit::YBEAM)); auto& yBeam = m_DataStructure.getDataRefAs(m_InputValues->CellAttributeMatrixPath.createChildPath(ebsdlib::H5Esprit::YBEAM)); - // Interleave 3 separate Euler angle arrays into 1x3 using bounded chunks + // Interleave 3 separate Euler angle arrays into a 3-component destination using + // bounded chunks. Applies degree-to-radian conversion in the local buffer before + // each bulk write via copyFromBuffer. { constexpr usize k_ChunkTuples = 65536; std::vector eulerChunk(k_ChunkTuples * 3); @@ -76,7 +96,8 @@ Result<> ReadH5EspritData::copyRawEbsdData(int index) } } - // Bulk copy single-component arrays directly from HDF5 reader buffers + // OOC-safe bulk copy of single-component arrays: each copyFromBuffer() writes the + // entire per-slice reader buffer at the correct offset in the volume-wide array. mad.getDataStoreRef().copyFromBuffer(offset, nonstd::span(m1, totalPoints)); nIndexBands.getDataStoreRef().copyFromBuffer(offset, nonstd::span(nIndBands, totalPoints)); phase.getDataStoreRef().copyFromBuffer(offset, nonstd::span(p1, totalPoints)); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.hpp index 8fb65b9e49..ba5d2b71d0 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ReadH5EspritData.hpp @@ -16,9 +16,16 @@ struct ORIENTATIONANALYSIS_EXPORT ReadH5EspritDataInputValues /** * @class ReadH5EspritData - * @brief This filter will read a single .h5 file into a new Image Geometry, allowing the immediate use of Filters on the data instead of having to generate the intermediate .h5ebsd file. + * @brief Algorithm that reads Bruker Esprit H5OINA EBSD data into an Image Geometry. + * + * Reads one slice at a time from the HDF5 file via the EbsdLib H5EspritReader, + * then copies each slice's data into the volume-wide DataStructure arrays. + * + * @section ooc_summary OOC Optimization Summary + * Same copyFromBuffer() bulk write approach as the other EBSD readers. Euler angles use + * chunked interleaving with optional degree-to-radian conversion. Single-component arrays + * are written in one bulk call per slice at the correct offset. */ - class ORIENTATIONANALYSIS_EXPORT ReadH5EspritData : public IEbsdOemReader { public: @@ -31,8 +38,16 @@ class ORIENTATIONANALYSIS_EXPORT ReadH5EspritData : public IEbsdOemReader operator()(); + /** + * @brief Copies raw EBSD data for one slice from the H5Esprit reader into the DataStructure. + * @param index Zero-based slice index, used to compute the tuple offset. + * @return Result<> indicating success or an error if pattern data is missing. + */ Result<> copyRawEbsdData(int index) override; private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp index cf2e31295e..e2e589f34e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.cpp @@ -25,6 +25,17 @@ RotateEulerRefFrame::RotateEulerRefFrame(DataStructure& dataStructure, const IFi RotateEulerRefFrame::~RotateEulerRefFrame() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Rotates all Euler angles in the dataset by a user-specified axis-angle + * rotation. Each Euler triplet is converted to an orientation matrix, multiplied + * by the rotation matrix, re-normalized, and converted back to Euler angles. + * + * OOC strategy: Replaced the parallel range-based approach with sequential + * chunked processing. Each 64K-tuple chunk is bulk-read from the DataStore via + * copyIntoBuffer, rotated in-place in the local buffer, then bulk-written back + * via copyFromBuffer. This is an in-place read-modify-write pattern on a single + * array. + */ Result<> RotateEulerRefFrame::operator()() { if(m_ShouldCancel) @@ -43,7 +54,8 @@ Result<> RotateEulerRefFrame::operator()() ebsdlib::OrientationMatrixDType omRot = ebsdlib::AxisAngleDType(axis[0], axis[1], axis[2], angle * nx::core::numbers::pi / 180.0).toOrientationMatrix(); OrientationUtilities::Matrix3dR rotMat = omRot.toEigenGMatrix(); - // Process in bounded chunks: read → rotate → write back + // Process in bounded 64K-tuple chunks: bulk-read, rotate locally, bulk-write. + // The buffer is reused across iterations to avoid allocation churn. constexpr usize k_ChunkTuples = 65536; std::vector buf(k_ChunkTuples * 3); @@ -54,6 +66,7 @@ Result<> RotateEulerRefFrame::operator()() return {}; } const usize count = std::min(k_ChunkTuples, totalTuples - startTup); + // Bulk-read this chunk of Euler angles (3 components per tuple) eulerStore.copyIntoBuffer(startTup * 3, nonstd::span(buf.data(), count * 3)); for(usize i = 0; i < count; i++) @@ -66,6 +79,7 @@ Result<> RotateEulerRefFrame::operator()() buf[i * 3 + 2] = eu[2]; } + // Bulk-write the rotated Euler angles back to the same DataStore location eulerStore.copyFromBuffer(startTup * 3, nonstd::span(buf.data(), count * 3)); } diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.hpp index 7becf79951..7efe58b140 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/RotateEulerRefFrame.hpp @@ -9,16 +9,38 @@ namespace nx::core { /** - * @brief The RotateEulerRefFrameInputValues struct + * @brief Input values for the RotateEulerRefFrame algorithm. */ struct ORIENTATIONANALYSIS_EXPORT RotateEulerRefFrameInputValues { - std::vector rotationAxis; - DataPath eulerAngleDataPath; + std::vector rotationAxis; ///< Rotation axis {x, y, z, angle_degrees} + DataPath eulerAngleDataPath; ///< Cell-level Float32 Euler angles (3 components, radians) }; /** - * @brief The RotateEulerRefFrame class + * @class RotateEulerRefFrame + * @brief Performs a passive rotation of Euler angles about a user-defined axis-angle pair. + * + * The reference frame is rotated, so the Euler angles are updated to represent + * the same physical orientation in the new frame. Each Euler angle triplet is + * converted to an orientation matrix, multiplied by the rotation matrix, and + * converted back to Euler angles. + * + * ## OOC Optimization (Major Rewrite) + * + * The original implementation used `ParallelDataAlgorithm` with a threaded + * worker that accessed the Euler angle array via `operator[]`. This caused + * severe performance degradation with OOC storage due to per-element virtual + * dispatch and random chunk access from multiple threads. + * + * The optimized implementation uses sequential chunked bulk I/O: + * - Euler angles are read in chunks of 65536 tuples via `copyIntoBuffer()`. + * - The rotation is applied to each tuple in the local buffer. + * - The modified buffer is written back via `copyFromBuffer()`. + * + * This in-place read-modify-write pattern is inherently sequential but + * provides excellent OOC throughput since each chunk is a single contiguous + * I/O operation. */ class ORIENTATIONANALYSIS_EXPORT RotateEulerRefFrame { @@ -31,7 +53,13 @@ class ORIENTATIONANALYSIS_EXPORT RotateEulerRefFrame RotateEulerRefFrame& operator=(const RotateEulerRefFrame&) = delete; // Copy Assignment Not Implemented RotateEulerRefFrame& operator=(RotateEulerRefFrame&&) = delete; // Move Assignment Not Implemented + /** + * @brief Executes the Euler angle rotation using chunked read-modify-write I/O. + * @return Result<> with any errors encountered during execution. + */ Result<> operator()(); + + /** @brief Returns whether the algorithm should cancel. */ bool shouldCancel() const; private: diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp index 0667a44e71..105f422e1e 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.cpp @@ -65,6 +65,21 @@ const std::atomic_bool& WriteGBCDGMTFile::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Writes GBCD data in GMT (Generic Mapping Tools) format for stereographic projection plotting. + * + * @section ooc_strategy OOC Strategy + * The GBCD array can be very large (5-dimensional, hundreds of MB). Rather than accessing + * individual bins via operator[] during the nested symmetry loops (which would cause + * devastating chunk thrashing on OOC stores), we: + * 1. Cache the crystal structures array locally (ensemble-level, tiny). + * 2. Extract only the phase-of-interest slice of the GBCD via a single copyIntoBuffer() + * call, bringing just the relevant subset into a local buffer. + * 3. All subsequent GBCD bin lookups in the O(nSym^2 * thetaPoints * phiPoints) inner + * loops use the local buffer, with zero OOC store access. + * + * @return Result<> indicating success or an error if file creation fails. + */ Result<> WriteGBCDGMTFile::operator()() { auto& gbcd = m_DataStructure.getDataRefAs(m_InputValues->GBCDArrayPath); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.hpp index 1aa1e7825d..ee54781ede 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDGMTFile.hpp @@ -11,17 +11,29 @@ namespace nx::core { +/** + * @brief Input values for the WriteGBCDGMTFile algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT WriteGBCDGMTFileInputValues { - int32 PhaseOfInterest; - VectorFloat32Parameter::ValueType MisorientationRotation; - FileSystemPathParameter::ValueType OutputFile; - DataPath GBCDArrayPath; - DataPath CrystalStructuresArrayPath; + int32 PhaseOfInterest; ///< Phase index to extract from the GBCD array. + VectorFloat32Parameter::ValueType MisorientationRotation; ///< Misorientation as [angle, axis_x, axis_y, axis_z]. + FileSystemPathParameter::ValueType OutputFile; ///< Path to the output GMT file. + DataPath GBCDArrayPath; ///< Path to the GBCD (5D) DataArray. + DataPath CrystalStructuresArrayPath; ///< Path to the CrystalStructures ensemble array. }; /** - * @class + * @class WriteGBCDGMTFile + * @brief Writes Grain Boundary Character Distribution (GBCD) data in GMT format for stereographic + * projection plotting. + * + * @section ooc_summary OOC Optimization Summary + * The GBCD array can be hundreds of MB (5-dimensional). Rather than accessing individual bins + * via operator[] during the O(nSym^2 * thetaPoints * phiPoints) inner loops, the algorithm + * caches only the phase-of-interest slice via a single copyIntoBuffer() call. Crystal + * structures (ensemble-level, tiny) are also cached locally. All subsequent lookups use + * the local buffers with zero OOC store access. */ class ORIENTATIONANALYSIS_EXPORT WriteGBCDGMTFile { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp index 65a9683e32..7a089f40e8 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.cpp @@ -25,6 +25,33 @@ const std::atomic_bool& WriteGBCDTriangleData::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Writes GBCD triangle data (grain boundary character distribution) to an ASCII file. + * + * Each line contains the Euler angles of the two grains adjacent to a triangle, + * the triangle normal, and the surface area. + * + * @section ooc_strategy OOC Strategy + * Three triangle-level arrays (faceLabels, faceNormals, faceAreas) are potentially + * very large (millions of triangles). Rather than reading each element via operator[] + * (which triggers chunk load/evict cycles on OOC stores), we: + * + * 1. Cache the Euler angles array locally via copyIntoBuffer(). This is feature-level + * data (one tuple per grain, typically thousands) and is small enough to hold entirely + * in memory. Grain IDs from faceLabels can map to arbitrary features, so caching + * the full array avoids random OOC lookups. + * + * 2. Process triangles in chunks of k_ChunkSize (8192). For each chunk: + * a. Bulk-read the chunk of faceLabels, faceNormals, and faceAreas via copyIntoBuffer(). + * b. Format all lines into a fmt::memory_buffer (pure in-memory string building). + * c. Write the entire buffer to disk in one outStream.write() call. + * + * This reduces OOC I/O from O(numTriangles) random accesses to O(numTriangles / k_ChunkSize) + * sequential bulk reads, and reduces file I/O from O(numTriangles) fprintf calls to + * O(numTriangles / k_ChunkSize) write calls. + * + * @return Result<> indicating success or an error if the output file cannot be opened. + */ Result<> WriteGBCDTriangleData::operator()() { auto& faceLabels = m_DataStructure.getDataRefAs(m_InputValues->SurfaceMeshFaceLabelsArrayPath); @@ -33,7 +60,9 @@ Result<> WriteGBCDTriangleData::operator()() auto& eulerAngles = m_DataStructure.getDataRefAs(m_InputValues->FeatureEulerAnglesArrayPath); usize numTriangles = faceAreas.getNumberOfTuples(); - // Cache eulerAngles locally — feature-level (indexed by grain ID, typically thousands) + // Cache eulerAngles locally -- feature-level (indexed by grain ID, typically thousands). + // This is small enough to hold entirely in memory and avoids random OOC lookups when + // grain IDs from faceLabels index into arbitrary positions. const usize numEulerElements = eulerAngles.getSize(); std::vector eulerCache(numEulerElements); eulerAngles.getDataStoreRef().copyIntoBuffer(0, nonstd::span(eulerCache.data(), numEulerElements)); @@ -49,7 +78,8 @@ Result<> WriteGBCDTriangleData::operator()() << "# Column 7-9: triangle normal\n" << "# Column 8: surface area\n"; - // Process triangles in chunks: bulk-read arrays, format into a string buffer, write once per chunk + // Process triangles in chunks: bulk-read arrays into local buffers, format into a + // string buffer, write once per chunk. This batches both OOC reads and file writes. constexpr usize k_ChunkSize = 8192; const auto& labelsStore = faceLabels.getDataStoreRef(); const auto& normalsStore = faceNormals.getDataStoreRef(); diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.hpp index 7c269a0212..13928252bb 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WriteGBCDTriangleData.hpp @@ -11,17 +11,27 @@ namespace nx::core { +/** + * @brief Input values for the WriteGBCDTriangleData algorithm. + */ struct ORIENTATIONANALYSIS_EXPORT WriteGBCDTriangleDataInputValues { - FileSystemPathParameter::ValueType OutputFile; - DataPath SurfaceMeshFaceLabelsArrayPath; - DataPath SurfaceMeshFaceNormalsArrayPath; - DataPath SurfaceMeshFaceAreasArrayPath; - DataPath FeatureEulerAnglesArrayPath; + FileSystemPathParameter::ValueType OutputFile; ///< Path to the output ASCII file. + DataPath SurfaceMeshFaceLabelsArrayPath; ///< Path to FaceLabels (2-component int32, grain IDs per face side). + DataPath SurfaceMeshFaceNormalsArrayPath; ///< Path to FaceNormals (3-component float64). + DataPath SurfaceMeshFaceAreasArrayPath; ///< Path to FaceAreas (1-component float64). + DataPath FeatureEulerAnglesArrayPath; ///< Path to FeatureEulerAngles (3-component float32, feature-level). }; /** - * @class + * @class WriteGBCDTriangleData + * @brief Writes grain boundary triangle data (Euler angles, normals, areas) to an ASCII file. + * + * @section ooc_summary OOC Optimization Summary + * Three face-level arrays (labels, normals, areas) are read in chunks via copyIntoBuffer() + * and formatted into a string buffer before writing. The feature-level Euler angles array + * is cached entirely in memory (small, one tuple per grain). This avoids per-element OOC + * access and reduces file I/O to one write per chunk. */ class ORIENTATIONANALYSIS_EXPORT WriteGBCDTriangleData { diff --git a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.hpp b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.hpp index 57dde873d3..4d687ac60f 100644 --- a/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.hpp +++ b/src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.hpp @@ -52,7 +52,16 @@ struct ORIENTATIONANALYSIS_EXPORT WritePoleFigureInputValues }; /** - * @class + * @class WritePoleFigure + * @brief Generates pole figure images from Euler angle data using Lambert or discrete projection. + * + * Supports multiple output formats (TIFF, BMP, PNG, JPG, PDF), layouts (horizontal, + * vertical, square), and optional intensity data output. The projection computation + * uses EbsdLib's ModifiedLambertProjection or a discrete stereographic projection. + * + * @note The OOC changes in this file are limited to type cleanup (size_t -> usize, + * float -> float32) and removing an unused include. The algorithm itself operates on + * EbsdLib's internal arrays which are always in-memory. */ class ORIENTATIONANALYSIS_EXPORT WritePoleFigure { diff --git a/src/Plugins/SimplnxCore/docs/AlignSectionsFeatureCentroidFilter.md b/src/Plugins/SimplnxCore/docs/AlignSectionsFeatureCentroidFilter.md index aa02d810ce..8b49fcb349 100644 --- a/src/Plugins/SimplnxCore/docs/AlignSectionsFeatureCentroidFilter.md +++ b/src/Plugins/SimplnxCore/docs/AlignSectionsFeatureCentroidFilter.md @@ -18,6 +18,19 @@ If the user elects to *Use Reference Slice*, then each section's centroid is shi The user can also decide to remove a *background shift* present in the sample. The process for this is to fit a line to the X and Y shifts along the Z-direction of the sample. The individual shifts are then modified to make the slope of the fit line be 0. Effectively, this process is trying to keep the top and bottom section of the sample fixed. Some combinations of sample geometry and internal features can result in this algorithm introducing a 'shear' in the sample and the *Linear Background Subtraction* will attempt to correct for this. +## Algorithm + +The algorithm proceeds in two phases: + +1. **Centroid Computation**: For each Z-slice (processed from top to bottom), the centroid of all "good" cells (as defined by the mask array) is computed by averaging the X and Y positions of all masked-in voxels. +2. **Shift Derivation**: Per-slice X/Y shifts are derived either progressively (each slice aligned to its predecessor) or relative to a single reference slice. Shifts are rounded to integer voxel increments. + +The base class then applies the computed shifts by physically reordering voxel data within each Z-slice for all cell-level data arrays. + +### Performance + +This filter is optimized for out-of-core (OOC) data storage. When the mask array resides on disk in chunked format, the algorithm automatically switches to a bulk I/O path that reads one complete Z-slice of mask data at a time via `copyIntoBuffer()`, rather than accessing each voxel individually. This eliminates the chunk load/evict cycle that would otherwise occur for every voxel, reducing I/O operations from O(total_voxels) to O(num_slices). + ## Optional Output Data The user can optionally have the shifts that are generated by the filter stored in various DataArrays in a new Attribute Matrix. diff --git a/src/Plugins/SimplnxCore/docs/ComputeArrayStatisticsFilter.md b/src/Plugins/SimplnxCore/docs/ComputeArrayStatisticsFilter.md index ee5ab4158c..c711823611 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeArrayStatisticsFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeArrayStatisticsFilter.md @@ -29,6 +29,10 @@ The input array may also be *standardized*, meaning that the array values will b Special operations occur for certain statistics if the supplied array is of type *bool* (for example, a mask array produced from threshold filters). The *length*, *minimum*, *maximum*, *median*, *mode*, and *summation* are computed as normal (although the resulting values may be platform dependent). The *mean* and *standard deviation* for a boolean array will be true if there are more instances of true in the array than false. If *Standardize Data* is chosen for a boolean array, no actual modifications will be made to the input. These operations for boolean inputs are chosen as a basic convention, and are not intended be representative of true boolean logic. +### Performance + +This filter is aware of out-of-core (OOC) data storage. When any input array is detected to be backed by an OOC DataStore, parallelization is automatically disabled to prevent concurrent random access to chunk-backed stores, which would cause severe performance degradation from chunk thrashing. + ## Destination Attribute Matrix The user must create a destination **Attribute Matrix** in which the computed statistics will be stored. DREAM3D-NX enforces a rule where any Attribute Matrix cannot contain another Attribute Matrix. With this in mind, the user should select a destination that is not itself an Attribute Matrix, such as the top level of a Geometry or the top level of the Data Structure itself. The user could have also created a group (using a previous filter) and use that group as the destination. diff --git a/src/Plugins/SimplnxCore/docs/ComputeBoundaryCellsFilter.md b/src/Plugins/SimplnxCore/docs/ComputeBoundaryCellsFilter.md index 9f3a965755..c6b9860484 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeBoundaryCellsFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeBoundaryCellsFilter.md @@ -17,6 +17,35 @@ This **Filter** determines, for each **Cell**, the number of neighboring **Cells |--|--| | ![Feature Ids](Images/ComputeBoundaryCellsInput.png) | ![Boundary Cells](Images/ComputeBoundaryCellsOutput.png) | +## Algorithm + +For each voxel in the image geometry, the filter counts how many of its 6 face-connected neighbors (front, back, left, right, up, down) belong to a different feature. The result is an Int8 array where each cell stores a value from 0 (all neighbors are the same feature) to 6 (all neighbors differ). + +Two optional behaviors modify the counting: + ++ **Include Volume Boundary**: When enabled, cells on the outer faces of the image geometry receive additional boundary counts for each face that touches the volume edge. Feature 0 cells on the boundary are excluded from this count. ++ **Ignore Feature Zero**: When enabled, neighbors with Feature ID = 0 are not counted as boundary faces. This is useful when Feature 0 represents background/empty space that should not be treated as a distinct feature. + +### In-Core Algorithm (Direct) + +The in-core variant iterates all voxels sequentially in Z-Y-X order. For each voxel, it uses pre-computed flat-index offsets to look up the 6 face neighbors directly via operator[] on the FeatureIds DataStore. This is a straightforward approach that works well when all data is resident in memory. + +### Out-of-Core Algorithm (Scanline) + +When the FeatureIds array is stored out-of-core in chunked format (e.g., loaded from a .dream3d file in OOC mode), the in-core algorithm's random neighbor lookups would trigger chunk load/evict cycles for every voxel, making it extremely slow. The Scanline variant avoids this by reading one complete Z-slice at a time using sequential bulk I/O. + +Three in-memory buffers hold adjacent Z-slices simultaneously: + ++ **prevSlice**: The Z-slice at z-1, needed for -Z neighbor lookups ++ **curSlice**: The Z-slice at z (the slice being processed) ++ **nextSlice**: The Z-slice at z+1, needed for +Z neighbor lookups + +Within a Z-slice, X and Y neighbor lookups are simple index arithmetic on the curSlice buffer. After processing a slice, the output is written in a single bulk operation, the window rotates forward, and the next Z-slice is loaded. This guarantees strictly sequential disk I/O. + +### Performance + +The in-core and out-of-core variants produce identical results. The algorithm dispatch is automatic: in-memory data uses the Direct path, and chunked/OOC data uses the Scanline path. The Scanline variant adds minimal memory overhead (3 Z-slices of int32 plus 1 Z-slice of int8), which is negligible compared to the full volume. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/ComputeEuclideanDistMapFilter.md b/src/Plugins/SimplnxCore/docs/ComputeEuclideanDistMapFilter.md index c96875a13a..b1a9ede956 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeEuclideanDistMapFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeEuclideanDistMapFilter.md @@ -20,6 +20,12 @@ This **Filter** calculates the distance of each **Cell** from the nearest **Feat 4. If the option *Calculate Manhattan Distance* is *false*, then the "city-block" distances are overwritten with the *Euclidean Distance* from the **Cell** to its internally tracked *nearest neighbor* **Cell** and stored in a *float* array instead of an *integer* array. +### Performance + +This filter is optimized for out-of-core (OOC) data storage. The entire Feature IDs array is bulk-read into a local buffer via `copyIntoBuffer()` at the start. Each distance map (grain boundary, triple junction, quadruple point) is also initialized and bulk-read into a local buffer. All boundary identification and iterative distance propagation operates on these local buffers with plain pointer access. Results are written back to the output DataStores via a single `copyFromBuffer()` call per map. This reduces OOC I/O operations from O(total_voxels * passes) to O(1) per map. + +The three distance maps (when enabled) are computed in parallel using `ParallelTaskAlgorithm`. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/ComputeFeatureCentroidsFilter.md b/src/Plugins/SimplnxCore/docs/ComputeFeatureCentroidsFilter.md index 77e435ffbb..414d64d667 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeFeatureCentroidsFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeFeatureCentroidsFilter.md @@ -8,6 +8,16 @@ Generic (Misc) This **Filter** calculates the *centroid* of each **Feature** by determining the average X, Y, and Z position of all the **Cells** belonging to the **Feature**. An *Is Periodic* option is available: when enabled, **Features** that extend beyond the boundary of the **Image Geometry** are treated as wrapping around to the opposite face, and centroids are computed accordingly. When *Is Periodic* is disabled, **Features** that intersect the outer surfaces of the sample will still have *centroids* calculated, but they will be *centroids* of the truncated part of the **Feature** that lies inside the sample. +## Algorithm + +The algorithm iterates over all voxels, accumulating each voxel's XYZ coordinate (computed from its flat index, the geometry origin, and spacing) into a per-feature Kahan sum. After all voxels are processed, each feature's centroid is the accumulated sum divided by the voxel count. + +If *Is Periodic* is enabled, a post-processing step checks whether any feature's bounding box spans the full extent of a dimension (indicating it wraps around a periodic boundary) and adjusts the centroid accordingly. + +### Performance + +This filter is optimized for out-of-core (OOC) data storage. The Feature IDs array is read in fixed-size chunks (64K tuples) via `copyIntoBuffer()` rather than accessing each voxel individually. All accumulation is performed in plain `std::vector` buffers (Kahan sums, compensators, voxel counts, XYZ ranges) to avoid per-element virtual dispatch through the DataStore interface. The final centroids are written to the output array in a single `copyFromBuffer()` call. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/ComputeFeatureClusteringFilter.md b/src/Plugins/SimplnxCore/docs/ComputeFeatureClusteringFilter.md index 018999d1aa..2924f0f385 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeFeatureClusteringFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeFeatureClusteringFilter.md @@ -16,6 +16,10 @@ This Filter determines the radial distribution function (RDF), as a histogram, o *Note:* Because the algorithm iterates over all the **Features**, each distance will be double counted. For example, the distance from **Feature** 1 to **Feature** 2 will be counted along with the distance from **Feature** 2 to **Feature** 1, which will be identical. +### Performance + +This filter has O(n^2) complexity in the number of features of the target phase. The feature-level arrays (phases, centroids) are accessed in the inner pairwise loop. For out-of-core (OOC) data, per-element virtual dispatch inside this quadratic loop would be prohibitively expensive. The algorithm bulk-reads the entire FeaturePhases and Centroids arrays into local `std::vector` caches via `copyIntoBuffer()` at the start. The RDF histogram is also accumulated into a local vector and written back to the output DataStore in a single `copyFromBuffer()` call after normalization. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/ComputeFeatureNeighborsFilter.md b/src/Plugins/SimplnxCore/docs/ComputeFeatureNeighborsFilter.md index ff02538bfb..e6f33c0ba1 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeFeatureNeighborsFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeFeatureNeighborsFilter.md @@ -15,6 +15,37 @@ This **Filter** determines, for each **Feature**, the number of other **Features While performing the above steps, the number of neighboring **Cells** with a different **Feature** owner than a given **Cell** is stored, which identifies whether a **Cell** lies on the surface/edge/corner of a **Feature** (i.e. the **Feature** boundary). Additionally, the surface area shared between each set of contiguous **Features** is calculated by tracking the number of times two neighboring **Cells** correspond to a contiguous **Feature** pair. The **Filter** also notes which **Features** touch the outer surface of the sample (this is obtained for "free" while performing the above algorithm). The **Filter** gives the user the option whether or not they want to store this additional information. +## Algorithm + +This filter has two algorithm implementations that are automatically selected at runtime based on how the input data is stored. The user does not need to choose between them. + +### In-Core Algorithm (Direct) + +When all input arrays reside in memory, the **Direct** algorithm is used. It employs compile-time dimension specialization (via C++ templates) to handle 0D, 1D, 2D, and 3D image geometries without runtime branching in the inner loops. + +Processing is split into two stages: + +1. **Boundary cells** (corners, edges, faces): Each voxel's face neighbors are checked with validity guards since boundary voxels do not have all 6 neighbors. +2. **Internal cells** (3D only): All 6 face neighbors are guaranteed to exist, so no validity checks are needed in the innermost loop. + +Surface area accumulation uses per-face area values computed from the geometry spacing, correctly handling non-cubic voxels. This fixes a bug present in DREAM3D 6.5 where all faces were assumed to have the same area. + +### Out-of-Core Algorithm (Scanline) + +When any input array is backed by chunked on-disk storage (out-of-core), the **Scanline** algorithm is used. Out-of-core data lives in compressed chunks on disk; random per-element access would trigger repeated chunk load/decompress/evict cycles ("chunk thrashing"), making the algorithm catastrophically slow. + +The Scanline algorithm avoids this by reading data one Z-slice at a time using bulk I/O, maintaining a rolling window of 3 Z-slices in memory: + +- **Previous slice** (z-1): Used for -Z neighbor lookups +- **Current slice** (z): The slice being processed +- **Next slice** (z+1): Used for +Z neighbor lookups + +Within each slice, +/-X and +/-Y neighbors are resolved by simple index arithmetic on the in-memory buffer. After processing a slice, the buffers rotate (prev gets cur, cur gets next, next loads z+2) and the BoundaryCells output is written back via bulk I/O. + +### Performance + +The in-core Direct algorithm accesses data through per-element getValue() calls, which are essentially pointer dereferences for in-memory data. The out-of-core Scanline algorithm uses sequential bulk I/O (copyIntoBuffer/copyFromBuffer), reading one Z-slice at a time. Memory usage is bounded to 3 Z-slices of FeatureIds plus 1 Z-slice of BoundaryCells, regardless of the total volume size. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/ComputeFeatureSizesFilter.md b/src/Plugins/SimplnxCore/docs/ComputeFeatureSizesFilter.md index f0eb4e4b54..96489b89fa 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeFeatureSizesFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeFeatureSizesFilter.md @@ -16,6 +16,16 @@ Note here that **Image Geometry** will always be faster than its equivalent **Re During the computation of the **Feature** sizes, the size of each individual **Element** is computed and stored in the corresponding **Geometry**. By default, these sizes are deleted after executing the **Filter** to save memory. If you wish to store the **Element** sizes, select the *Generate Missing Element Sizes* option. The sizes will be stored within the **Geometry** definition itself, not as a separate **Attribute Array**. +## Algorithm + +For **Image Geometry**, the algorithm counts voxels per feature, then multiplies by the uniform voxel volume (product of spacings) to obtain volumes. For 2D image geometries (one dimension equals 1), areas and equivalent circular diameters are computed instead. + +For **Rectilinear Grid Geometry**, where each cell can have a different volume, the algorithm reads both the Feature IDs and element sizes in lockstep and uses Kahan summation to accurately accumulate per-feature volumes. + +### Performance + +This filter is optimized for out-of-core (OOC) data storage. The Feature IDs array (and element sizes for Rectilinear Grid) is read in fixed-size chunks (64K tuples) via `copyIntoBuffer()`. Per-feature voxel counts and volumes are accumulated in plain `std::vector` buffers. This chunked approach reduces the number of OOC I/O operations from O(total_voxels) to O(total_voxels / 64K). + ## Image Geometry Additional Considerations A typical Image Stack *(an `Image Geometry` that contains 3 dimensions greater than 1)* conceptually consists of a series of 2D images stacked on top of one another to create a 3D object, thus it functions in 3D space as expected. This means it produces **Volumes** and **Equivalent Spherical Diameters**. diff --git a/src/Plugins/SimplnxCore/docs/ComputeKMedoidsFilter.md b/src/Plugins/SimplnxCore/docs/ComputeKMedoidsFilter.md index 2f3f4f931c..b8f34b06d1 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeKMedoidsFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeKMedoidsFilter.md @@ -54,6 +54,35 @@ A clustering algorithm can be considered a kind of segmentation; this implementa This **Filter** will store the medoids for the final clusters within the created **Attribute Matrix**. +## Algorithm + +This filter has two algorithm implementations that are automatically selected at runtime based on how the input data is stored. The user does not need to choose between them. + +### In-Core Algorithm (Direct) + +When all input arrays reside in memory, the **Direct** algorithm is used. It accesses array elements via direct per-element operator[] calls, which are optimal for in-memory data (essentially pointer dereferences). + +The algorithm performs the standard Voronoi iteration: + +1. **Initialize**: Randomly select k data points as initial medoids +2. **Assign clusters**: For each data point, compute the distance to all k medoids and assign it to the nearest +3. **Optimize medoids**: For each cluster, find the member that minimizes the total intra-cluster distance +4. **Repeat** steps 2-3 until medoids stop changing (convergence) + +### Out-of-Core Algorithm (Scanline) + +When any input array is backed by chunked on-disk storage (out-of-core), the **Scanline** algorithm is used. Out-of-core data lives in compressed chunks on disk; each per-element operator[] access would trigger a chunk load/decompress/evict cycle ("chunk thrashing"), making the iterative algorithm extremely slow. + +The Scanline algorithm avoids this with three key optimizations: + +- **Medoid caching**: The medoids array is small (k points), so it is cached entirely in a local buffer before each cluster assignment pass, eliminating k * N per-element OOC reads per iteration. +- **Chunked cluster assignment**: The input data and cluster IDs are read and written in aligned 64K-tuple chunks via bulk I/O (copyIntoBuffer/copyFromBuffer). All distance computations for each chunk are done in memory. +- **Per-cluster member scanning**: During medoid optimization, cluster member indices are collected by scanning the cluster IDs in chunks. Pairwise distances within each cluster are computed using single-tuple bulk reads. Peak memory is proportional to the largest cluster, not the total data size. + +### Performance + +The in-core Direct algorithm is faster for in-memory data due to the lower overhead of operator[] access. The out-of-core Scanline algorithm converts random per-element access into sequential bulk I/O, which is essential for data stored on disk in compressed chunks. Both produce identical clustering results. + % Auto generated parameter table will be inserted here ## References diff --git a/src/Plugins/SimplnxCore/docs/ComputeSurfaceAreaToVolumeFilter.md b/src/Plugins/SimplnxCore/docs/ComputeSurfaceAreaToVolumeFilter.md index 52117bd1fb..981503fef4 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeSurfaceAreaToVolumeFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeSurfaceAreaToVolumeFilter.md @@ -17,11 +17,35 @@ This **Filter** determines whether a **Feature** touches an outer *Surface* of t + Any cell location is xmin, xmax, ymin, ymax, zmin or zmax + Any cell has **Feature ID = 0** as a neighbor. -## Algorithm Details +## Algorithm -- First, all the boundary **Cells** are found for each **Feature**. -- Next, the surface area for each face that is in contact with a different **Feature** is totalled as long as that neighboring *featureId* is > 0. -- This number is divided by the volume of each **Feature**, calculated by taking the number of **Cells** of each **Feature** and multiplying by the volume of a **Cell**. +The filter computes the surface-area-to-volume ratio for each feature in two phases: + +**Phase 1 -- Surface area accumulation**: For each voxel in the image geometry, examine its 6 face-connected neighbors. When a neighbor belongs to a different feature (and the neighbor's Feature ID > 0), the area of the shared face is added to the current feature's surface area total. The face area depends on which axis the face is normal to: + ++ Z-normal faces (shared by +/-Z neighbors): spacing.x * spacing.y ++ Y-normal faces (shared by +/-Y neighbors): spacing.y * spacing.z ++ X-normal faces (shared by +/-X neighbors): spacing.z * spacing.x + +**Phase 2 -- Ratio and sphericity**: For each feature, divide the accumulated surface area by the feature's volume (number of cells * voxel volume). If sphericity is requested, it is computed as: sphericity = (pi^(1/3) * (6V)^(2/3)) / SA, where a perfect sphere has sphericity = 1.0. + +### In-Core Algorithm (Direct) + +The in-core variant iterates all voxels in Z-Y-X order and uses pre-computed flat-index offsets to look up the 6 face neighbors directly via operator[] on the FeatureIds DataStore. Surface area is accumulated into a local vector (since multiple voxels contribute to each feature), and the final ratio is written to the output array. + +### Out-of-Core Algorithm (Scanline) + +When the FeatureIds array is stored out-of-core in chunked format, the in-core algorithm's scattered neighbor lookups would trigger chunk thrashing. The Scanline variant reads one complete Z-slice at a time using sequential bulk I/O, maintaining a 3-slice rolling window: + ++ **prevSlice**: Z-slice at z-1, needed for -Z neighbor lookups ++ **curSlice**: Z-slice at z (being processed) ++ **nextSlice**: Z-slice at z+1, needed for +Z neighbor lookups + +Within a Z-slice, X and Y neighbors are simple index offsets within curSlice. After the voxel scan, the feature-level NumCells array is also bulk-read into a local cache, the ratio and optional sphericity are computed locally, and the results are bulk-written back. This ensures zero random-access operator[] calls on any OOC DataStore. + +### Performance + +The in-core and out-of-core variants produce identical results. The algorithm dispatch is automatic based on the storage type of the FeatureIds array. ### WARNING - Aliasing diff --git a/src/Plugins/SimplnxCore/docs/ComputeSurfaceFeaturesFilter.md b/src/Plugins/SimplnxCore/docs/ComputeSurfaceFeaturesFilter.md index 395152e694..34ed01ffd6 100644 --- a/src/Plugins/SimplnxCore/docs/ComputeSurfaceFeaturesFilter.md +++ b/src/Plugins/SimplnxCore/docs/ComputeSurfaceFeaturesFilter.md @@ -34,6 +34,36 @@ If the structure/data is actually 2D, then the dimension that is planar is not c | ![ComputeSurfaceFeatures_Cylinder](Images/ComputeSurfaceFeatures_Cylinder.png) | ![ComputeSurfaceFeatures_Square](Images/ComputeSurfaceFeatures_Square.png) | | Example showing features touching Feature ID=0 (Black voxels) "Mark Feature 0 Neighbors" is **ON** | Example showing features touching the outer surface of the bounding box | +## Algorithm + +The filter examines every voxel in the image geometry and determines whether the feature owning that voxel qualifies as a "surface feature." A feature is marked as a surface feature the first time any of its voxels meets the surface criteria; once marked, subsequent voxels of that feature are skipped (short-circuit optimization). + +A voxel qualifies its feature as a surface feature if: + +1. The voxel is located on the outer boundary of the image geometry (x, y, or z is at its minimum or maximum value), **OR** +2. Any of the voxel's face neighbors has Feature ID = 0 (when **Mark Feature 0 Neighbors** is enabled). + +For 2D geometries (where one dimension has size 1), the degenerate dimension is collapsed and only the 4 in-plane neighbors are checked. + +### In-Core Algorithm (Direct) + +The in-core variant uses separate code paths for 3D and 2D geometries. For 3D, it iterates all voxels in Z-Y-X order and checks 6 face neighbors via operator[] on the FeatureIds DataStore. For 2D, it determines which dimension is degenerate and iterates the non-degenerate plane, checking 4 neighbors. This approach works well when all data is resident in memory. + +### Out-of-Core Algorithm (Scanline) + +When the FeatureIds array is stored out-of-core in chunked format, the in-core algorithm's random neighbor lookups would trigger chunk load/evict cycles. The Scanline variant reads one complete native Z-slice at a time using sequential bulk I/O, maintaining a 3-slice rolling window (prevSlice, curSlice, nextSlice). + +For 3D geometries, the neighbor lookups map directly to the rolling window buffers. For 2D geometries, the algorithm still iterates the native Z-Y-X grid but remaps coordinates to the logical 2D plane: + ++ **Degenerate Z** (most common): All data fits in a single Z-slice; all 4 neighbors come from curSlice. ++ **Degenerate X or Y**: The remapped-Y direction maps to the native Z axis, so +/-Y neighbors come from the adjacent Z-slice buffers. + +The feature-level SurfaceFeatures output is cached in a local vector during processing and bulk-written once at the end, avoiding per-voxel random writes to the OOC store. + +### Performance + +The in-core and out-of-core variants produce identical results. The algorithm dispatch is automatic based on the storage type of the FeatureIds array. Memory overhead for the Scanline variant is 3 Z-slices of int32 (for the rolling window) plus a small feature-level vector. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/CropImageGeometryFilter.md b/src/Plugins/SimplnxCore/docs/CropImageGeometryFilter.md index 06955dbc50..1f67302f40 100644 --- a/src/Plugins/SimplnxCore/docs/CropImageGeometryFilter.md +++ b/src/Plugins/SimplnxCore/docs/CropImageGeometryFilter.md @@ -70,6 +70,14 @@ In this example the user is going to define the crop using physical coordinates User may note that the way the bounds are determined are affected by the origin and spacing, so be sure to take these into account when supplying coordinate bounds for the crop. +## Algorithm + +The crop operation copies voxel data from the source geometry's bounding region into a new (smaller) geometry. For each cell-level data array, the data is copied one X-row at a time using bulk `copyIntoBuffer()` and `copyFromBuffer()` calls. + +### Performance + +This filter is optimized for out-of-core (OOC) data storage. The original implementation copied data element-by-element in a triple-nested loop (Z, Y, X), causing a DataStore chunk operation per voxel per component. The optimized implementation copies entire X-rows at a time, reducing the number of I/O operations from O(voxels * components) to O(Z_range * Y_range), a factor-of-X_range improvement. + ## Renumber Features It is possible with this **Filter** to fully remove **Features** from the volume, possibly resulting in consistency errors if more **Filters** process the data in the pipeline. If the user selects to *Renumber Features* then the *Feature Ids* array will be adjusted so that all **Features** are continuously numbered starting from 1. The user should decide if they would like their **Features** renumbered or left alone (in the case where the cropped output is being compared to some larger volume). diff --git a/src/Plugins/SimplnxCore/docs/DBSCANFilter.md b/src/Plugins/SimplnxCore/docs/DBSCANFilter.md index 968178fe47..1ae1f2fea5 100644 --- a/src/Plugins/SimplnxCore/docs/DBSCANFilter.md +++ b/src/Plugins/SimplnxCore/docs/DBSCANFilter.md @@ -172,6 +172,33 @@ The *Distance Metric* parameter provides the following choices: The inclusion of randomness in this algorithm is solely to attempt to reduce bias from starting cluster. Three parse order options are available: *Low Density First* (deterministic, no seed needed), *Random* (non-deterministic, uses a time-based seed), and *Seeded Random* (deterministic, uses a user-supplied seed value for reproducibility). Low Density First produced identical results faster in our test cases, but the random initialization is truest to the well known DBSCAN algorithm. +## Algorithm + +This filter has two algorithm implementations that are automatically selected at runtime based on how the input data is stored. The user does not need to choose between them. + +### In-Core Algorithm (Direct) + +When all input arrays reside in memory, the **Direct** algorithm is used. It accesses array elements via direct per-element operator[] calls, which are optimal for in-memory data. + +The grid-based DBSCAN algorithm proceeds in three phases: + +1. **Grid construction**: The input array is scanned to determine coordinate bounds, create a regular grid with cell side length epsilon / sqrt(dimensions), and bin each data point into a grid cell. Bit-packed adjacency tables are built for fast neighbor grid queries. +2. **Clustering**: Core grid cells (those with >= minPoints) are identified and processed according to the selected parse order. Adjacent grid cells are merged into the same cluster if any pair of points between them has distance < epsilon. Non-core cells are then expanded into nearby clusters. +3. **Labeling**: Final cluster IDs are written to the output array. Points in unassigned cells become outliers (cluster 0). + +### Out-of-Core Algorithm (Scanline) + +When any input array is backed by chunked on-disk storage (out-of-core), the **Scanline** algorithm is used. Out-of-core data lives in compressed chunks on disk; per-element operator[] access during the multi-pass grid construction and random-access distance checks would trigger repeated chunk load/decompress/evict cycles. + +The Scanline algorithm addresses this with two key modifications: + +- **Chunked grid construction**: Instead of per-element operator[] access during the 2-3 passes over the input array (bounds detection, binning, cell filling), the Scanline variant reads data in sequential 64K-tuple chunks via copyIntoBuffer(). This converts millions of per-element random accesses into thousands of sequential bulk reads. +- **On-demand grid cell reads**: During distance checks between adjacent grid cells (canMerge), the Direct variant randomly indexes into the full input array for every point in both cells. The Scanline variant instead bulk-reads all coordinate data for each grid cell into a local buffer, then performs all pairwise distance computations entirely in memory. + +### Performance + +The in-core Direct algorithm is faster for in-memory data due to the lower overhead of operator[] access. The out-of-core Scanline algorithm converts random per-element access into sequential bulk I/O, which is essential for data stored on disk in compressed chunks. The clustering and cluster-expansion phases operate on the in-memory grid index in both variants, so their performance is identical. Both produce the same clustering results. + % Auto generated parameter table will be inserted here ## References diff --git a/src/Plugins/SimplnxCore/docs/ErodeDilateBadDataFilter.md b/src/Plugins/SimplnxCore/docs/ErodeDilateBadDataFilter.md index 4e1c68abef..df69a14a45 100644 --- a/src/Plugins/SimplnxCore/docs/ErodeDilateBadDataFilter.md +++ b/src/Plugins/SimplnxCore/docs/ErodeDilateBadDataFilter.md @@ -54,6 +54,33 @@ The *Operation* parameter selects which morphological operation to apply: - **Dilate [0]**: Grows bad data regions by one **Cell** per iteration. Any **Cell** neighboring a bad **Cell** has its *Feature Id* changed to 0. - **Erode [1]**: Shrinks bad data regions by one **Cell** per iteration. Each bad **Cell** is assigned the *Feature Id* of the majority of its neighbors. +## Algorithm + +This filter performs iterative morphological erosion or dilation on "bad" voxels (cells with FeatureId == 0) within an ImageGeom grid. + +### Erosion + +For each bad voxel, the algorithm examines its 6 face-connected neighbors and tallies the FeatureIds of any good (non-zero) neighbors. The bad voxel is then assigned the FeatureId that appears most frequently among its good neighbors (a "majority vote"). If there is a tie, one of the tied FeatureIds is chosen. This process shrinks bad-data regions by one cell per iteration. + +### Dilation + +For each bad voxel, the algorithm examines its 6 face-connected neighbors. Any good neighbor adjacent to the bad voxel has its FeatureId set to 0, effectively growing the bad-data region outward by one cell per iteration. + +In both cases, all sibling data arrays in the same Attribute Matrix (except those in the user's ignored list) are updated to match the FeatureId changes, so the data remains consistent. + +### Iteration + +The operation is repeated for the user-specified number of iterations. Each iteration makes a full pass over the volume. Because each pass modifies the data, subsequent iterations see the cumulative effect of all prior passes. + +### Performance + +This algorithm is optimized for both in-memory and out-of-core (OOC) data stores. When data resides on disk in chunked format, random voxel access can cause expensive chunk load/evict cycles. The implementation avoids this by: + +- **Sequential Z-slice processing**: The volume is scanned one Z-slice at a time, which aligns with typical chunk boundaries and avoids random access patterns. +- **3-slice rolling window**: Three adjacent Z-slices of FeatureIds are held in memory simultaneously, allowing face-neighbor lookups without hitting the data store for each voxel. +- **Deferred bulk writes**: Data modifications are batched per Z-slice and written back in bulk, minimizing the number of I/O operations. +- **O(sliceSize) memory**: Per-slice mark arrays replace a full-volume neighbor array, keeping peak memory proportional to a single Z-slice rather than the entire volume. + ## WARNING: Feature Data Will Become Invalid By modifying the cell level data, any feature data that was previously computed will most likely be invalid at this point. Filters that compute feature level data should be rerun to ensure accurate final results from your pipeline. diff --git a/src/Plugins/SimplnxCore/docs/ErodeDilateCoordinationNumberFilter.md b/src/Plugins/SimplnxCore/docs/ErodeDilateCoordinationNumberFilter.md index 2e11f06787..05d9763e19 100644 --- a/src/Plugins/SimplnxCore/docs/ErodeDilateCoordinationNumberFilter.md +++ b/src/Plugins/SimplnxCore/docs/ErodeDilateCoordinationNumberFilter.md @@ -23,6 +23,32 @@ Gone* parameter, which will continue to run until no **Cells** fail the original |--------------------------------------|--------------------------------------| | ![](Images/ErodeDilateCoordinationNumber_Before.png) | ![](Images/ErodeDilateCoordinationNumber_After.png) | +## Algorithm + +For each voxel on a good/bad boundary (where "good" means FeatureId > 0 and "bad" means FeatureId == 0), the algorithm counts how many of its 6 face-connected neighbors belong to the opposite class. This count is the voxel's **coordination number**. + +A high coordination number means a voxel is mostly surrounded by the opposite type and is likely a boundary artifact or noise. For example, a single bad voxel completely surrounded by good voxels has a coordination number of 6. + +### Processing Steps + +1. For each boundary voxel, compute the coordination number by counting opposite-type face neighbors. +2. Among those opposite-type neighbors, identify the most common FeatureId. +3. If the coordination number meets or exceeds the user's threshold, mark the voxel to be replaced by the most common neighbor's data. +4. After scanning the entire volume, apply all marked replacements. + +If **Loop Until Gone** is enabled, the algorithm repeats this process until no voxels exceed the coordination number threshold. Each pass may create new boundary conditions that expose previously acceptable voxels, so multiple passes can be necessary to fully smooth the interface. + +All sibling data arrays in the same Attribute Matrix (except those in the user's ignored list) are updated along with the FeatureIds to maintain data consistency. + +### Performance + +This algorithm is optimized for both in-memory and out-of-core (OOC) data stores. When data resides on disk in chunked format, random voxel access can cause expensive chunk load/evict cycles. The implementation avoids this by: + +- **Sequential Z-slice processing**: The volume is scanned one Z-slice at a time, aligning with typical chunk boundaries. +- **3-slice rolling window**: Three adjacent Z-slices of FeatureIds are held in memory for face-neighbor lookups without per-voxel store access. +- **Conditional deferred writes**: Only voxels whose coordination number meets the threshold are transferred, and writes are batched per Z-slice. +- **O(sliceSize) memory**: Per-slice mark and coordination arrays replace full-volume arrays, keeping peak memory proportional to a single Z-slice. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/ErodeDilateMaskFilter.md b/src/Plugins/SimplnxCore/docs/ErodeDilateMaskFilter.md index 8f81cd62e1..c363b2a941 100644 --- a/src/Plugins/SimplnxCore/docs/ErodeDilateMaskFilter.md +++ b/src/Plugins/SimplnxCore/docs/ErodeDilateMaskFilter.md @@ -32,6 +32,30 @@ The *Operation* parameter selects which morphological operation to apply to the - **Dilate [0]**: Expands the masked (true) regions by one **Cell** per iteration. Any **Cell** neighboring a false **Cell** is changed to true. - **Erode [1]**: Shrinks the masked (true) regions by one **Cell** per iteration. True **Cells** that have at least one false neighbor are changed to false. +## Algorithm + +This filter performs iterative morphological erosion or dilation directly on a boolean mask array within an ImageGeom grid. Unlike the Erode/Dilate Bad Data filter, this filter operates only on the mask and does not propagate changes to sibling data arrays. + +### Processing Steps + +For each iteration, the algorithm scans every voxel in the volume: + +1. Identify false (unmasked) voxels. +2. For each false voxel, examine its 6 face-connected neighbors (optionally restricted to specific axes by the user). +3. **Dilation**: If any neighbor is true (masked), set the current false voxel to true. This grows the masked region outward. +4. **Erosion**: If any neighbor is true, set that neighbor to false. This shrinks the masked region inward. + +A dual-buffer approach ensures that reads and writes do not interfere within a single iteration: the original mask state is read from one buffer while modifications are accumulated in a separate copy. + +### Performance + +This algorithm is optimized for both in-memory and out-of-core (OOC) data stores. When data resides on disk in chunked format, random voxel access can cause expensive chunk load/evict cycles. The implementation avoids this by: + +- **Sequential Z-slice processing**: The volume is scanned one Z-slice at a time, aligning with typical chunk boundaries. +- **3-slice dual rolling window**: Two sets of three Z-slice buffers (read and write) are maintained in memory, allowing face-neighbor lookups and modification tracking without per-voxel store access. +- **Deferred bulk writes**: Modified slices are written back to the store in bulk after each Z-layer completes, minimizing I/O operations. +- **uint8 intermediary for bool**: Because std::vector uses bit-packing, uint8 buffers are used for the rolling window with conversion during I/O. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/IdentifySampleFilter.md b/src/Plugins/SimplnxCore/docs/IdentifySampleFilter.md index 333175f841..667d7262df 100644 --- a/src/Plugins/SimplnxCore/docs/IdentifySampleFilter.md +++ b/src/Plugins/SimplnxCore/docs/IdentifySampleFilter.md @@ -46,6 +46,40 @@ When *Process Data Slice-By-Slice* is enabled, the *Slice-By-Slice Plane* parame - **XZ [1]**: Processes the volume slice by slice along the Y axis, scanning each XZ plane independently. - **YZ [2]**: Processes the volume slice by slice along the X axis, scanning each YZ plane independently. +## Algorithm + +This filter identifies the largest connected region of "good" voxels (the sample) and marks all other voxels as "bad." Two algorithm paths are available, selected automatically based on the underlying data storage. + +### In-Core Path (BFS) + +When data resides entirely in memory, a **breadth-first search (BFS)** flood fill is used: + +1. Iterate through all voxels and, for each unvisited "good" voxel, start a BFS that explores all 6-connected face neighbors. +2. Track the largest connected component found — this is identified as the sample. +3. Set all "good" voxels **not** in the largest component to "bad." +4. If **Fill Holes** is enabled, run a second BFS pass over "bad" voxels: any connected region of "bad" voxels that does not touch the volume boundary is filled back to "good." + +BFS is efficient for in-memory data because the queue-driven traversal has excellent cache locality when all data fits in RAM. + +### Out-of-Core Path (CCL) + +When any input array uses chunked on-disk storage (out-of-core / OOC), BFS would cause **chunk thrashing** — each random queue-driven access may load and evict entire disk chunks, making the algorithm 100–1000× slower. Instead, a **connected component labeling (CCL)** approach is used: + +1. **Scanline labeling**: Iterate voxels sequentially (Z → Y → X), assigning provisional labels to "good" voxels. When two labeled regions are found to be connected (same row or adjacent Z-slice), their labels are merged using a **Union-Find** data structure. +2. **Global resolution**: Flatten the Union-Find tree so every provisional label maps to its final root label. Count the size of each component. +3. **Classification**: The largest component is kept as the sample; all other "good" voxels are set to "bad." +4. **Hole filling** (if enabled): A second CCL pass identifies connected components of "bad" voxels and fills any that do not touch the volume boundary. + +The sequential access pattern aligns with OOC chunk layout, reading each chunk at most once. + +### Slice-By-Slice Mode + +When **Process Data Slice-By-Slice** is enabled, both the in-core and OOC paths use a shared `IdentifySampleSliceBySliceFunctor` that processes individual 2D slices. Since a single 2D slice is small enough to fit in memory, BFS is always safe and efficient for this mode. For the **YZ plane**, a batched read strategy reads each Z-slice once for a batch of X-columns, reducing HDF5 I/O operations by ~10× compared to reading per-column. + +### Performance + +The CCL path provides 10–100× speedup over BFS for large datasets stored out-of-core. For in-memory datasets, BFS is typically faster due to lower overhead. The dispatch is automatic — no user configuration is needed. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/MultiThresholdObjectsFilter.md b/src/Plugins/SimplnxCore/docs/MultiThresholdObjectsFilter.md index 2e07d2da61..86b35f4af9 100644 --- a/src/Plugins/SimplnxCore/docs/MultiThresholdObjectsFilter.md +++ b/src/Plugins/SimplnxCore/docs/MultiThresholdObjectsFilter.md @@ -16,6 +16,31 @@ It is possible to set custom values for both the TRUE and FALSE values that will **NOTE**: If custom TRUE/FALSE values are chosen, then using the resulting mask array in any other filters that require a mask array will break those other filters. This is because most other filters that require a mask array make the assumption that the true/false values are 1/0. +## Algorithm + +This filter has two algorithm implementations that are automatically selected at runtime based on how the input data is stored. The user does not need to choose between them. + +### In-Core Algorithm (Direct) + +When all input arrays reside in memory, the **Direct** algorithm is used. For each threshold condition, it reads the input array via per-element access and compares every element against the threshold value. The results are stored in a temporary vector, then merged into the output mask using AND/OR logic. + +### Out-of-Core Algorithm (Scanline) + +When any input array is backed by chunked on-disk storage (out-of-core), the **Scanline** algorithm is used. Out-of-core data lives in compressed chunks on disk; per-element access would trigger repeated chunk load/decompress/evict cycles ("chunk thrashing"). + +The Scanline algorithm processes data in fixed-size chunks (64K tuples at a time): + +1. Read a chunk of the input array via bulk I/O (copyIntoBuffer) +2. Apply the threshold comparison to produce a chunk-sized result buffer +3. For the first threshold condition, write results directly to the output mask via bulk I/O +4. For subsequent conditions, read the current output chunk, merge using AND/OR logic, and write back + +This approach replaces the Direct variant's per-element reads with sequential bulk reads, and reduces temporary memory from O(n) to O(64K) per threshold condition. + +### Performance + +The in-core Direct algorithm is faster for in-memory data. The out-of-core Scanline algorithm converts random per-element access into sequential bulk I/O and reduces peak memory usage. For a dataset with 100 million tuples, the Scanline variant uses approximately 64 KB of temporary memory per threshold condition instead of approximately 100 MB. Both produce identical output. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/QuickSurfaceMeshFilter.md b/src/Plugins/SimplnxCore/docs/QuickSurfaceMeshFilter.md index 1eca069532..d76b708a00 100644 --- a/src/Plugins/SimplnxCore/docs/QuickSurfaceMeshFilter.md +++ b/src/Plugins/SimplnxCore/docs/QuickSurfaceMeshFilter.md @@ -71,6 +71,40 @@ Each triangle that is created will have an 2 component attribute called `Face La side of the triangle. If one of the triangles represents the border of the virtual box then one of the FaceLables will have a value of -1. +## Algorithm + +This filter uses a dispatch mechanism to select the optimal algorithm implementation based on the storage type of the input arrays. + +### In-Core Algorithm (Direct) + +When all input arrays are backed by in-memory storage, the **QuickSurfaceMeshDirect** algorithm is used. This is the original implementation that accesses the FeatureIds array via direct element indexing. + +The algorithm proceeds in three phases: + +1. **Problem Voxel Correction** (optional): Iteratively examines every 2x2x2 block of voxels to detect diagonal-conflict configurations that would produce non-manifold mesh geometry. Conflicting voxels are randomly reassigned to a neighbor's FeatureId using a seeded RNG for reproducibility. Up to 20 correction iterations are performed. + +2. **Node and Triangle Counting**: A single pass over all voxels counts the number of unique mesh vertices (nodes) and boundary triangles. For each voxel, the algorithm checks whether the FeatureId differs from the +X, +Y, and +Z neighbors. Volume boundary faces also produce triangles. A mapping array of size (xP+1) x (yP+1) x (zP+1) assigns sequential vertex IDs to active dual-grid corners. + +3. **Mesh Generation**: A second pass writes vertex coordinates, triangle connectivity, face labels, and node types. Face labels ensure the smaller FeatureId is always in component[0], with -1 used for exterior boundary faces. Each vertex is classified by how many features share it (2=interior face, 3=triple line, 4=quad point, +10 for boundary vertices). + +### Out-of-Core Algorithm (Scanline) + +When any input array uses chunked out-of-core (OOC) storage, the **QuickSurfaceMeshScanline** algorithm is selected automatically. This variant produces identical output but avoids random-access reads that would cause chunk thrashing on disk-backed data stores. + +Key optimizations: + +- **Z-slice bulk I/O**: FeatureIds are read one Z-slice at a time (xP x yP elements) via `copyIntoBuffer()` instead of per-element reads. At most two adjacent Z-slices are buffered simultaneously. + +- **Rolling node-plane buffers**: Instead of the O(volume) node mapping array used by the Direct variant, two node-plane buffers of size O((xP+1) x (yP+1)) each are maintained and swapped after each Z-slice. This reduces memory from O(volume) to O(slice). + +- **Buffered output writes**: Triangle connectivity and face labels are accumulated in per-slice buffers and flushed via `copyFromBuffer()`. Vertex coordinates are buffered for all nodes and flushed once at the end. + +- **Dirty-flag write-back**: During problem voxel correction, modified Z-slices are tracked with dirty flags and only written back if they were actually changed. + +### Performance + +The in-core (Direct) variant is fastest for datasets that fit in memory. The out-of-core (Scanline) variant avoids the 100-1000x performance penalty that would occur from chunk thrashing on OOC datasets, at the cost of slightly more complex bookkeeping. Both variants produce bit-identical output. + ## Notes The Quick Mesh algorithm is very crude and naive in its implementation. This filter diff --git a/src/Plugins/SimplnxCore/docs/RegularGridSampleSurfaceMeshFilter.md b/src/Plugins/SimplnxCore/docs/RegularGridSampleSurfaceMeshFilter.md index 5e20d2b89a..a131da6485 100644 --- a/src/Plugins/SimplnxCore/docs/RegularGridSampleSurfaceMeshFilter.md +++ b/src/Plugins/SimplnxCore/docs/RegularGridSampleSurfaceMeshFilter.md @@ -15,7 +15,7 @@ The *Use existing geometry* parameter controls how the target **Image Geometry** - **Create geometry [0]**: Creates a new **Image Geometry** with user-specified dimensions, spacing, and origin for sampling the surface mesh. - **Use existing geometry [1]**: Uses a pre-existing **Image Geometry** from the data structure as the sampling grid. -### Algorithm +## Algorithm The sampling is performed using the following scanline rasterization approach: @@ -24,6 +24,15 @@ The sampling is performed using the following scanline rasterization approach: 3. Sort the crossings by X-coordinate and walk left to right across the voxels, toggling the current **Feature Id** at each boundary crossing 4. Assign the current **Feature Id** to each voxel in the *Feature Ids* output array +### Performance + +This filter is optimized for out-of-core (OOC) data storage in two ways: + +1. **Input pre-loading**: All triangle geometry data (faces, vertices, face labels) is bulk-read into contiguous memory buffers via `copyIntoBuffer()` at algorithm start. Worker threads then operate on plain memory pointers with no per-element virtual dispatch overhead. +2. **Thread-safe output**: Each Z-slice worker rasterizes into a thread-local buffer, then writes results to the output DataArray via a mutex-protected `copyFromBuffer()` call. This ensures correct and efficient bulk I/O with OOC DataStore implementations. + +Z-slices are processed in parallel using `ParallelTaskAlgorithm`, with per-triangle Z-range pre-computation enabling fast rejection of triangles that don't intersect a given slice. + ### Face Labels / Part Numbers The **Face Labels/Part Numbers** input array specifies which **Features** (or parts) border each **Triangle** face. This array accepts any integer data type and supports two formats: diff --git a/src/Plugins/SimplnxCore/docs/ReplaceElementAttributesWithNeighborValuesFilter.md b/src/Plugins/SimplnxCore/docs/ReplaceElementAttributesWithNeighborValuesFilter.md index 6511f9eb40..54ac600328 100644 --- a/src/Plugins/SimplnxCore/docs/ReplaceElementAttributesWithNeighborValuesFilter.md +++ b/src/Plugins/SimplnxCore/docs/ReplaceElementAttributesWithNeighborValuesFilter.md @@ -83,6 +83,32 @@ scan point was not indexed. By using the *Error* value from the data file we can Note the large areas of unindexed pixels in the original image (black pixels) and how they are all filled in. The filter can act much like a generic "flood fill" image processing algorithm if used improperly. +## Algorithm + +This filter iteratively replaces voxel data that fails a user-defined threshold comparison with data from the best-scoring face neighbor. + +### Processing Steps + +For each pass over the volume: + +1. For each voxel, compare its value in the selected array against the threshold using the chosen comparison operator (less-than or greater-than). +2. If the voxel fails the comparison (e.g., confidence index < 0.1), examine its 6 face-connected neighbors. +3. Among neighbors that pass the threshold, find the one with the best value (highest for less-than mode, lowest for greater-than mode). +4. Mark the failing voxel to be replaced by that neighbor's data. +5. After each Z-slice is scanned, apply all replacements across every array in the Attribute Matrix (not just the comparison array). + +If **Loop Until Gone** is enabled, the algorithm repeats until no voxels fail the threshold. Each pass can improve neighbors of previously failing voxels, allowing the cleanup to propagate inward from good-data boundaries. + +### Performance + +This algorithm is optimized for both in-memory and out-of-core (OOC) data stores. When data resides on disk in chunked format, random voxel access can cause expensive chunk load/evict cycles. The implementation avoids this by: + +- **Sequential Z-slice processing**: The volume is scanned one Z-slice at a time, aligning with typical chunk boundaries. +- **3-slice rolling window**: Three adjacent Z-slices of the comparison array are held in typed memory buffers, allowing face-neighbor value lookups without per-voxel store access. +- **Immediate per-slice transfer**: Because replacement marks always point to face neighbors (within one Z-slice), each slice can be committed immediately after its scan completes, keeping writes sequential. +- **O(sliceSize) memory**: A single per-slice mark array replaces a full-volume neighbor array, keeping peak memory proportional to one Z-slice. +- **Type-dispatched inner loop**: The comparison and transfer logic is templated on the input array's element type, avoiding virtual dispatch overhead in the tight inner loop. + % Auto generated parameter table will be inserted here ## Example Pipelines diff --git a/src/Plugins/SimplnxCore/docs/RequireMinimumSizeFeaturesFilter.md b/src/Plugins/SimplnxCore/docs/RequireMinimumSizeFeaturesFilter.md index d3a1c185b3..9ee89bd828 100644 --- a/src/Plugins/SimplnxCore/docs/RequireMinimumSizeFeaturesFilter.md +++ b/src/Plugins/SimplnxCore/docs/RequireMinimumSizeFeaturesFilter.md @@ -10,6 +10,20 @@ This **Filter** removes **Features** that have a total number of **Cells** below The **Filter** can be run in a mode where the minimum number of neighbors is applied to a single **Ensemble**. The user can select to apply the minimum to one specific **Ensemble**. +## Algorithm + +The algorithm operates in two phases: + +1. **Feature Removal**: Features with fewer voxels than the threshold have their voxels' Feature IDs set to -1 (invalid). This is done using chunked bulk I/O (64K tuples per chunk). +2. **Gap Filling**: The resulting "holes" (voxels with Feature ID = -1) are iteratively filled by majority voting among each hole voxel's 6 face-neighbors. Each iteration assigns the most-voted valid neighbor's Feature ID. This repeats until no holes remain. + +### Performance + +This filter is optimized for out-of-core (OOC) data storage in both phases: + +- **Feature Removal**: Feature IDs are read and written in 64K-tuple chunks via `copyIntoBuffer()` / `copyFromBuffer()`. Only chunks that contain modifications are written back. +- **Gap Filling**: A rolling 3-slice buffer holds the previous, current, and next Z-slices of Feature IDs in memory. All 6 face-neighbor reads come from these local buffers rather than per-element OOC DataStore access. Changed voxels are tracked in a compact list, and only those voxels have their data arrays updated, rather than scanning all voxels for each data array. + ## WARNING: Feature Data Will Become Invalid By modifying the cell level data, any feature data that was previously computed will most likely be invalid at this point. Filters that compute feature level data should be rerun to ensure accurate final results from your pipeline. diff --git a/src/Plugins/SimplnxCore/docs/ScalarSegmentFeaturesFilter.md b/src/Plugins/SimplnxCore/docs/ScalarSegmentFeaturesFilter.md index ab536a2667..1c101dbe1b 100644 --- a/src/Plugins/SimplnxCore/docs/ScalarSegmentFeaturesFilter.md +++ b/src/Plugins/SimplnxCore/docs/ScalarSegmentFeaturesFilter.md @@ -20,6 +20,21 @@ After all the **Features** have been identified, an **Attribute Matrix** is crea If the data is specified as **Periodic**, the segmentation will check if features wrap around geometry bounds in a tileable fashion. If any such features are detected, the filter will throw a warning that centroid data may be incorrect. +## Algorithm + +The filter has two execution paths selected automatically at runtime: + +- **In-core (DFS flood fill)**: The classic depth-first search algorithm described above. Each voxel is visited via element-by-element access through a typed comparator. +- **Out-of-core (Connected-Component Labeling)**: A slice-by-slice CCL algorithm that processes data Z-slice at a time. This path is activated automatically when the data resides in an out-of-core (OOC) DataStore. + +Both paths produce identical segmentation results. After segmentation, the Feature Attribute Matrix is resized, the Active array is initialized, and Feature IDs are optionally randomized. + +### Performance + +When operating on out-of-core data, the CCL path uses a rolling 2-slot buffer system. Before processing each Z-slice, the algorithm bulk-reads the scalar input and mask arrays for that slice into contiguous in-memory buffers. All voxel comparisons then read from these buffers rather than the underlying disk-backed DataStore, eliminating chunk load/evict cycles. Two buffer slots are maintained simultaneously (current and previous slice) because the CCL algorithm must compare voxels across adjacent slices. + +All scalar types are converted to `float64` in the buffer for uniform comparison, so a single comparison code path handles all input data types. + ### Neighbor Scheme The *Neighbor Scheme* parameter provides the following choices: diff --git a/src/Plugins/SimplnxCore/docs/SurfaceNetsFilter.md b/src/Plugins/SimplnxCore/docs/SurfaceNetsFilter.md index 658ca935b9..ea5c3feec9 100644 --- a/src/Plugins/SimplnxCore/docs/SurfaceNetsFilter.md +++ b/src/Plugins/SimplnxCore/docs/SurfaceNetsFilter.md @@ -66,6 +66,46 @@ Each triangle that is created will have an 2 component attribute called `Face La side of the triangle. If one of the triangles represents the border of the virtual box then one of the FaceLables will have a value of -1. +## Algorithm + +This filter uses a dispatch mechanism to select the optimal algorithm implementation based on the storage type of the input arrays. + +### In-Core Algorithm (Direct) + +When all input arrays are backed by in-memory storage, the **SurfaceNetsDirect** algorithm is used. This delegates to the MMSurfaceNet library, which is a C++ implementation of the Surface Nets algorithm from Frisken (2022). + +The algorithm proceeds in six phases: + +1. **Build Surface Net**: The MMSurfaceNet library constructs a padded grid (dimX+2, dimY+2, dimZ+2) and classifies every cell by examining its 8 corner labels. Cells where not all corners have the same FeatureId are "surface cells" and receive a mesh vertex at the cell center. This reads the entire FeatureIds array via direct element access. + +2. **Smoothing** (optional): Iterative Laplacian-like relaxation moves each vertex toward the average of its face-connected neighbors, clamped to stay within `MaxDistanceFromVoxel` of the cell center. The `RelaxationFactor` controls the blending between current and average position. + +3. **Vertex Transformation**: Converts cell-local coordinates (where 0.5 = cell center) to world coordinates using the ImageGeom origin and spacing. + +4. **Triangle Counting**: First pass over surface vertices, checking 3 edges per cell (BackBottom, LeftBottom, LeftBack) for feature boundary crossings. Each crossing produces a quad (4 vertices) that becomes 2 triangles. + +5. **Triangle Generation**: Second pass that writes triangle connectivity and face labels. Quads are triangulated using the diagonal that minimizes total triangle area, reducing self-intersections. + +6. **Winding Repair** (optional): Fixes inconsistent triangle orientations. + +### Out-of-Core Algorithm (Scanline) + +When any input array uses chunked out-of-core (OOC) storage, the **SurfaceNetsScanline** algorithm is selected automatically. This variant reimplements the entire Surface Nets algorithm without the MMSurfaceNet library, using only O(surface) memory instead of O(volume). + +Key optimizations: + +- **Z-slice bulk I/O**: FeatureIds are read two Z-slices at a time via `copyIntoBuffer()` with a rolling ping-pong buffer. Each cell's 8 corner labels are resolved from the two buffered slices using a `cornerLabel()` helper. + +- **O(surface) cell storage**: Instead of allocating one Cell per padded voxel (the O(volume) MMCellMap), only surface cells are stored in a vector of `VertexInfo` structs plus a hash map for O(1) neighbor lookups. For typical datasets, the surface is O(n^{2/3}) vs O(n) for the full volume. + +- **Self-contained smoothing**: The relaxation is performed entirely on the O(surface) vertex data using neighbor lookups through the hash map, without needing the full MMCellMap. + +- **Buffered output writes**: All triangle connectivity, face labels, vertex coordinates, and node types are accumulated in local buffers and flushed via `copyFromBuffer()` in bulk. + +### Performance + +The in-core (Direct) variant is fastest for datasets that fit in memory, leveraging the optimized MMSurfaceNet library. The out-of-core (Scanline) variant avoids the O(volume) memory allocation and per-element reads that would cause chunk thrashing on OOC datasets. For a 500x500x500 dataset with ~1% surface cells, the Scanline variant uses roughly 100x less memory for cell classification. Both variants produce identical output. + ## Notes This filter should be used in place of the "QuickMesh Surface Filter". diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp index f3f31ffe90..529b09d7da 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.cpp @@ -23,6 +23,12 @@ AlignSectionsFeatureCentroid::AlignSectionsFeatureCentroid(DataStructure& dataSt // ----------------------------------------------------------------------------- AlignSectionsFeatureCentroid::~AlignSectionsFeatureCentroid() noexcept = default; +// ----------------------------------------------------------------------------- +/** + * @brief Entry point: delegates to the base-class AlignSections::execute() which + * calls findShifts() to compute per-slice shifts, then applies them to all cell + * data arrays by physically reordering voxels within each Z-slice. + */ // ----------------------------------------------------------------------------- Result<> AlignSectionsFeatureCentroid::operator()() { @@ -35,9 +41,17 @@ Result<> AlignSectionsFeatureCentroid::operator()() return execute(gridGeom.getDimensions(), m_InputValues->ImageGeometryPath); } +// ----------------------------------------------------------------------------- +/** + * @brief Computes per-slice X/Y centroid shifts. Dispatches to findShiftsOoc() + * when the mask array is out-of-core to avoid per-element chunk thrashing; + * otherwise uses the in-memory MaskCompare path with per-element isTrue() calls. + */ // ----------------------------------------------------------------------------- Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, std::vector& yShifts) { + // Check if OOC dispatch is needed before creating MaskCompare (which would + // eagerly load the entire array for some store types). { const auto& maskCheck = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); if(ForceOocAlgorithm() || IsOutOfCore(maskCheck)) @@ -234,12 +248,24 @@ Result<> AlignSectionsFeatureCentroid::findShifts(std::vector& xShifts, s } // ----------------------------------------------------------------------------- -// OOC-optimized findShifts: bulk-reads mask per Z-slice instead of per-element -// isTrue() calls, eliminating OOC chunk thrashing in the centroid computation. +/** + * @brief OOC-optimized shift computation. Instead of per-element MaskCompare::isTrue() + * calls (which trigger a chunk load per voxel for OOC stores), this method reads + * one complete Z-slice of mask data at a time via copyIntoBuffer(). The centroid + * computation then iterates over the in-memory buffer with zero OOC overhead. + * + * Memory usage: one XY-slice of uint8 mask data (dims[0] * dims[1] bytes), plus + * a temporary bool[] buffer of the same size when the mask is boolean-typed. + * + * The shift calculation logic after centroid computation is identical to the + * in-core path (without the StoreAlignmentShifts diagnostic storage, which is + * handled separately at the end). + */ // ----------------------------------------------------------------------------- Result<> AlignSectionsFeatureCentroid::findShiftsOoc(std::vector& xShifts, std::vector& yShifts) { - // Get raw mask store for bulk reads + // Obtain typed DataStore pointers for bulk reads. Only uint8 and bool masks + // are supported; other types produce an error. const auto& maskArray = m_DataStructure.getDataRefAs(m_InputValues->MaskArrayPath); const AbstractDataStore* maskUInt8StorePtr = nullptr; const AbstractDataStore* maskBoolStorePtr = nullptr; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp index 134d054bfa..c67638bc5e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/AlignSectionsFeatureCentroid.hpp @@ -11,24 +11,43 @@ namespace nx::core { +/** + * @struct AlignSectionsFeatureCentroidInputValues + * @brief Holds all user-configured parameters for the AlignSectionsFeatureCentroid algorithm. + */ struct SIMPLNXCORE_EXPORT AlignSectionsFeatureCentroidInputValues { - DataPath ImageGeometryPath; - DataPath MaskArrayPath; + DataPath ImageGeometryPath; ///< Path to the ImageGeom whose Z-slices will be aligned. + DataPath MaskArrayPath; ///< Path to the boolean/uint8 mask array identifying "good" cells. - bool UseReferenceSlice; - int32 ReferenceSlice; + bool UseReferenceSlice; ///< If true, align all slices to a single reference slice rather than progressively. + int32 ReferenceSlice; ///< Z-index of the reference slice (only used when UseReferenceSlice is true). - bool StoreAlignmentShifts; - DataPath AlignmentAMPath; - DataPath SlicesArrayPath; - DataPath RelativeShiftsArrayPath; - DataPath CumulativeShiftsArrayPath; - DataPath CentroidsArrayPath; + bool StoreAlignmentShifts; ///< If true, write per-slice shift diagnostics to output arrays. + DataPath AlignmentAMPath; ///< Path to the Attribute Matrix that will hold the diagnostic arrays. + DataPath SlicesArrayPath; ///< Output: slice indices (uint32, 2-component). + DataPath RelativeShiftsArrayPath; ///< Output: per-slice relative X/Y shifts (int64, 2-component). + DataPath CumulativeShiftsArrayPath; ///< Output: per-slice cumulative X/Y shifts (int64, 2-component). + DataPath CentroidsArrayPath; ///< Output: per-slice XY centroids (float32, 2-component). }; /** - * @class + * @class AlignSectionsFeatureCentroid + * @brief Aligns Z-slices of an ImageGeom by computing the centroid of masked + * (good) cells in each slice and shifting slices so their centroids align. + * + * The algorithm iterates over Z-slices from top to bottom, computes the weighted + * centroid of all "good" cells in each slice (as defined by the mask array), then + * derives per-slice X/Y shifts that bring consecutive (or reference-relative) + * centroids into alignment. Shifts are rounded to integer voxel increments. + * + * @section ooc_optimization Out-of-Core Optimization + * When the mask array resides in an out-of-core (OOC) DataStore, per-element + * MaskCompare::isTrue() calls would trigger a chunk load/evict cycle for every + * voxel, making the centroid loop extremely slow. The OOC path (findShiftsOoc) + * instead bulk-reads one full Z-slice of mask data at a time via copyIntoBuffer(), + * then iterates over the in-memory buffer. This converts O(N) random chunk + * accesses into O(Z) sequential bulk reads, where Z is the number of slices. */ class SIMPLNXCORE_EXPORT AlignSectionsFeatureCentroid : public AlignSections { @@ -41,24 +60,39 @@ class SIMPLNXCORE_EXPORT AlignSectionsFeatureCentroid : public AlignSections AlignSectionsFeatureCentroid& operator=(const AlignSectionsFeatureCentroid&) = delete; AlignSectionsFeatureCentroid& operator=(AlignSectionsFeatureCentroid&&) noexcept = delete; + /** + * @brief Executes the alignment algorithm: computes centroids, derives shifts, + * and applies them via the base-class AlignSections::execute() method. + * @return Result<> indicating success or error. + */ Result<> operator()(); protected: /** - * @brief This method finds the slice to slice shifts and should be implemented by subclasses - * @param xShifts - * @param yShifts - * @return Whether the x and y shifts were successfully found + * @brief Computes per-slice X/Y shifts by comparing centroids of masked cells. + * + * Dispatches to findShiftsOoc() when the mask array is out-of-core; otherwise + * uses the in-memory MaskCompare path for per-element access. + * @param xShifts Output vector of X shifts per Z-slice (in voxel units). + * @param yShifts Output vector of Y shifts per Z-slice (in voxel units). + * @return Result<> indicating success or error. */ Result<> findShifts(std::vector& xShifts, std::vector& yShifts) override; private: + /** + * @brief OOC-optimized shift computation that bulk-reads the mask array one + * Z-slice at a time via copyIntoBuffer(), avoiding per-element chunk thrashing. + * @param xShifts Output vector of X shifts per Z-slice (in voxel units). + * @param yShifts Output vector of Y shifts per Z-slice (in voxel units). + * @return Result<> indicating success or error. + */ Result<> findShiftsOoc(std::vector& xShifts, std::vector& yShifts); - DataStructure& m_DataStructure; - const AlignSectionsFeatureCentroidInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const AlignSectionsFeatureCentroidInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp index a2c0a168db..81613ab59c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp @@ -18,6 +18,14 @@ using namespace nx::core; namespace { // ----------------------------------------------------------------------------- +/** + * @brief Checks whether all arrays in the set are backed by in-memory DataStores. + * Used to decide if parallelization is safe: OOC stores are not thread-safe for + * concurrent random access, so parallelization must be disabled when any array + * resides out-of-core. Uses IDataStore::getStoreType() for an explicit check + * rather than the legacy getDataFormat() string comparison. + */ +// ----------------------------------------------------------------------------- bool CheckArraysInMemory(const nx::core::IParallelAlgorithm::AlgorithmArrays& arrays) { if(arrays.empty()) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.hpp index a754fd4d90..fc448ea923 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.hpp @@ -55,7 +55,16 @@ struct SIMPLNXCORE_EXPORT ComputeArrayStatisticsInputValues }; /** - * @class + * @class ComputeArrayStatistics + * @brief Computes a configurable set of statistical measures (length, min, max, + * mean, median, mode, standard deviation, summation, unique value count) for a + * scalar array, optionally grouped by Feature/Ensemble ID. + * + * @section ooc_note Out-of-Core Awareness + * The algorithm detects whether input arrays are out-of-core (OOC) by checking + * IDataStore::getStoreType(). When OOC arrays are detected, parallelization is + * disabled to prevent concurrent random access to chunk-backed stores, which + * would cause severe performance degradation from chunk thrashing. */ class SIMPLNXCORE_EXPORT ComputeArrayStatistics { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp index 1f6ad3a5c3..2d6a355fa7 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.cpp @@ -8,6 +8,23 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeBoundaryCells -- Dispatcher +// +// This file implements the thin dispatch layer for the ComputeBoundaryCells +// algorithm. No algorithm logic lives here; the sole responsibility is to +// inspect the storage type of the FeatureIds array and forward execution to +// either ComputeBoundaryCellsDirect (in-core) or ComputeBoundaryCellsScanline +// (out-of-core), via the DispatchAlgorithm template. +// +// The FeatureIds array is the critical input because it is a cell-level array +// with one entry per voxel. For a 500x500x500 volume that is ~125 million +// int32 values. When stored out-of-core in chunked format, random-access reads +// through operator[] trigger chunk load/evict cycles that make the algorithm +// catastrophically slow. The Scanline variant avoids this by using sequential +// bulk I/O (copyIntoBuffer/copyFromBuffer) one Z-slice at a time. +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeBoundaryCells::ComputeBoundaryCells(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues) : m_DataStructure(dataStructure) @@ -27,6 +44,16 @@ const std::atomic_bool& ComputeBoundaryCells::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Inspects the FeatureIds array's storage type and dispatches to the + * appropriate algorithm variant. + * + * The dispatch decision is made by DispatchAlgorithm, which checks: + * 1. ForceInCoreAlgorithm() -- test override, always selects Direct + * 2. AnyOutOfCore({featureIdsArray}) -- runtime detection of chunked storage + * 3. ForceOocAlgorithm() -- test override, forces Scanline + * 4. Default -- selects Direct (in-core) + */ Result<> ComputeBoundaryCells::operator()() { auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp index 954173dde4..1c44c0452b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCells.hpp @@ -9,21 +9,54 @@ namespace nx::core { +/** + * @struct ComputeBoundaryCellsInputValues + * @brief Holds all user-configurable parameters for the ComputeBoundaryCells algorithm. + * + * These values are extracted from the filter's parameter map and passed through + * the dispatcher to whichever algorithm variant (Direct or Scanline) is selected. + */ struct SIMPLNXCORE_EXPORT ComputeBoundaryCellsInputValues { - bool IgnoreFeatureZero; - bool IncludeVolumeBoundary; - DataPath ImageGeometryPath; - DataPath FeatureIdsArrayPath; - DataPath BoundaryCellsArrayName; + bool IgnoreFeatureZero; ///< When true, neighbors with FeatureId == 0 are not counted as boundary faces. + bool IncludeVolumeBoundary; ///< When true, cells on the edge of the image geometry volume contribute extra boundary counts. + DataPath ImageGeometryPath; ///< Path to the ImageGeom that defines grid dimensions. + DataPath FeatureIdsArrayPath; ///< Path to the cell-level Int32 FeatureIds array. + DataPath BoundaryCellsArrayName; ///< Path where the output Int8 boundary-cell-count array will be stored. }; /** - * @class + * @class ComputeBoundaryCells + * @brief Dispatcher that selects between the in-core (Direct) and out-of-core (Scanline) + * boundary-cell counting algorithms at runtime. + * + * This class does not contain any algorithm logic itself. Its operator()() inspects + * the storage backing of the FeatureIds array and calls + * `DispatchAlgorithm(...)`. + * + * **Algorithm overview**: For each voxel in the image geometry, count how many of its + * 6 face-connected neighbors belong to a different feature. The result is an Int8 array + * where each cell stores its boundary face count (0-6). + * + * **Dispatch rules** (see AlgorithmDispatch.hpp): + * - If all input arrays are backed by in-memory DataStore, the Direct variant is used. + * - If any input array uses out-of-core (chunked/Zarr) storage, the Scanline variant + * is used to avoid random-access chunk thrashing. + * - Global test-override flags (ForceOocAlgorithm, ForceInCoreAlgorithm) can override + * the automatic detection for unit testing purposes. + * + * @see ComputeBoundaryCellsDirect, ComputeBoundaryCellsScanline, DispatchAlgorithm */ class SIMPLNXCORE_EXPORT ComputeBoundaryCells { public: + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all arrays and geometries. + * @param mesgHandler Handler for sending progress/info messages back to the UI. + * @param shouldCancel Atomic flag checked periodically to support user cancellation. + * @param inputValues User-configured parameters for the algorithm. + */ ComputeBoundaryCells(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeBoundaryCellsInputValues* inputValues); ~ComputeBoundaryCells() noexcept; @@ -32,15 +65,24 @@ class SIMPLNXCORE_EXPORT ComputeBoundaryCells ComputeBoundaryCells& operator=(const ComputeBoundaryCells&) = delete; ComputeBoundaryCells& operator=(ComputeBoundaryCells&&) noexcept = delete; + /** + * @brief Dispatches to the appropriate algorithm variant (Direct or Scanline) + * based on whether the FeatureIds array uses out-of-core storage. + * @return Result<> indicating success or any errors encountered. + */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag. + * @return Const reference to the atomic cancellation boolean. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; ///< User-configured algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Atomic flag for cooperative cancellation. + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress and informational messages. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp index 9bbcea5def..817e59bf62 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.cpp @@ -8,6 +8,23 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeBoundaryCellsDirect -- In-Core Algorithm +// +// Counts, for each voxel, how many of its 6 face-connected neighbors belong to +// a different feature. The output is an Int8 array with values in [0, 6]. +// +// Data access pattern: This variant reads FeatureIds and writes BoundaryCells +// via operator[], which is efficient when the underlying DataStore is a +// contiguous in-memory buffer (pointer dereference). It uses pre-computed +// neighbor index offsets from NeighborUtilities.hpp so that each neighbor +// lookup is a single addition + array index. +// +// This is NOT suitable for out-of-core data because the 6-neighbor access +// pattern is spatially scattered (especially the +/-Z neighbors, which are +// dimX*dimY elements apart), causing chunk thrashing on chunked stores. +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeBoundaryCellsDirect::ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeBoundaryCellsInputValues* inputValues) @@ -23,11 +40,27 @@ ComputeBoundaryCellsDirect::~ComputeBoundaryCellsDirect() noexcept = default; // ----------------------------------------------------------------------------- /** - * @brief Counts boundary faces per voxel using direct Z-Y-X iteration. - * In-core path: iterates all voxels sequentially, checking 6 face neighbors. + * @brief Counts boundary faces per voxel using direct in-memory array indexing. + * + * The algorithm proceeds as follows: + * 1. Retrieve image geometry dimensions and compute flat-index offsets for the + * 6 face neighbors (-Z, -Y, -X, +X, +Y, +Z). + * 2. For each voxel in Z-Y-X order: + * a. Optionally count volume-boundary faces (if the voxel sits on the edge + * of the image geometry and IncludeVolumeBoundary is enabled). + * b. For each of the 6 face neighbors, check if the neighbor is inside the + * volume and belongs to a different feature. If so, increment the count. + * c. Store the count in the BoundaryCells output array. + * + * The IgnoreFeatureZero flag controls whether neighbors with FeatureId == 0 + * are counted as boundary faces. When true, only neighbors with FeatureId > 0 + * that differ from the current voxel's feature contribute to the count. + * + * @return Result<> indicating success (empty errors vector) or cancellation. */ Result<> ComputeBoundaryCellsDirect::operator()() { + // -- Step 1: Retrieve geometry dimensions and set up neighbor offset tables -- const auto& imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const SizeVec3 udims = imageGeometry.getDimensions(); std::array dims = { @@ -36,9 +69,14 @@ Result<> ComputeBoundaryCellsDirect::operator()() static_cast(udims[2]), }; + // Get direct references to the underlying data stores. Safe here because + // the dispatcher only selects this variant when stores are in-memory. auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); auto& boundaryCellsStore = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsArrayName)->getDataStoreRef(); + // Pre-compute the flat-index offsets for the 6 face neighbors. For a voxel + // at flat index i, the -Z neighbor is at i - (dimX*dimY), the -Y neighbor + // is at i - dimX, etc. This avoids recomputing these offsets per voxel. std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); @@ -46,6 +84,10 @@ Result<> ComputeBoundaryCellsDirect::operator()() int8 onSurf = 0; int64 neighborPoint = 0; + // When IgnoreFeatureZero is true, ignoreFeatureZeroVal == 0, so the condition + // `neighborFeature > 0` filters out feature-0 neighbors. When false, + // ignoreFeatureZeroVal == -1, so `neighborFeature > -1` allows feature 0 + // to count as a boundary neighbor. int ignoreFeatureZeroVal = 0; if(!m_InputValues->IgnoreFeatureZero) { @@ -55,8 +97,10 @@ Result<> ComputeBoundaryCellsDirect::operator()() int64 kStride = 0; int64 jStride = 0; + // -- Step 2: Main Z-Y-X iteration over all voxels -- for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { + // Check for user cancellation once per Z-slice to avoid overhead if(m_ShouldCancel) { return {}; @@ -72,6 +116,11 @@ Result<> ComputeBoundaryCellsDirect::operator()() feature = featureIdsStore[voxelIndex]; if(feature >= 0) { + // -- Step 2a: Volume boundary contribution -- + // If the voxel is on the edge of the image geometry and the user + // enabled IncludeVolumeBoundary, count each edge face. The dim > 2 + // guard avoids counting boundaries for trivially thin dimensions. + // Feature 0 voxels on the boundary are excluded (reset to 0). if(m_InputValues->IncludeVolumeBoundary) { if(dims[0] > 2 && (xIdx == 0 || xIdx == dims[0] - 1)) @@ -93,7 +142,11 @@ Result<> ComputeBoundaryCellsDirect::operator()() } } - // Loop over the 6 face neighbors of the voxel + // -- Step 2b: Check 6 face neighbors -- + // computeValidFaceNeighbors returns a bool[6] indicating which + // neighbors are inside the volume (boundary voxels have fewer + // valid neighbors). For each valid neighbor, compare its feature + // ID to the current voxel's feature ID. std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); for(const auto& faceIndex : faceNeighborInternalIdx) { @@ -103,12 +156,15 @@ Result<> ComputeBoundaryCellsDirect::operator()() } neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; + // Count this face as a boundary if the neighbor belongs to a + // different feature AND passes the feature-zero filter. if(featureIdsStore[neighborPoint] != feature && featureIdsStore[neighborPoint] > ignoreFeatureZeroVal) { onSurf++; } } } + // -- Step 2c: Store the boundary count for this voxel -- boundaryCellsStore[voxelIndex] = onSurf; } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp index b2ed65ffe4..ffc099faff 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsDirect.hpp @@ -11,13 +11,38 @@ struct ComputeBoundaryCellsInputValues; /** * @class ComputeBoundaryCellsDirect - * @brief In-core algorithm for ComputeBoundaryCells. Preserves the original sequential - * Z-Y-X voxel iteration with face-neighbor boundary counting. Selected by DispatchAlgorithm - * when all input arrays are backed by in-memory DataStore. + * @brief In-core (direct memory access) algorithm for counting boundary cells. + * + * This is the traditional algorithm that uses operator[] to read FeatureIds and write + * BoundaryCells directly through the DataStore abstraction. It iterates all voxels in + * Z-Y-X order and, for each voxel, checks its 6 face-connected neighbors using + * pre-computed index offsets from NeighborUtilities.hpp. + * + * **When this variant is selected**: DispatchAlgorithm selects this class when all + * input arrays are backed by contiguous in-memory DataStore (i.e., not chunked/OOC). + * With in-memory data, operator[] is a simple pointer dereference, so random neighbor + * lookups are essentially free. + * + * **Why a separate OOC variant exists**: When FeatureIds is stored out-of-core in + * chunked format (e.g., Zarr/HDF5 chunks), every operator[] call may trigger a chunk + * load from disk. The 6-neighbor access pattern means up to 7 chunk loads per voxel + * (the voxel itself plus its neighbors), which causes catastrophic "chunk thrashing" + * and can slow the algorithm by 100-1000x. The Scanline variant avoids this by + * reading entire Z-slices sequentially with copyIntoBuffer(). + * + * @see ComputeBoundaryCellsScanline for the OOC-optimized variant. + * @see ComputeBoundaryCells for the dispatcher. */ class SIMPLNXCORE_EXPORT ComputeBoundaryCellsDirect { public: + /** + * @brief Constructs the in-core boundary cell counter. + * @param dataStructure The DataStructure containing FeatureIds and BoundaryCells arrays. + * @param mesgHandler Handler for progress/info messages. + * @param shouldCancel Atomic flag for cooperative cancellation. + * @param inputValues Algorithm parameters (geometry path, array paths, flags). + */ ComputeBoundaryCellsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeBoundaryCellsInputValues* inputValues); ~ComputeBoundaryCellsDirect() noexcept; @@ -26,13 +51,22 @@ class SIMPLNXCORE_EXPORT ComputeBoundaryCellsDirect ComputeBoundaryCellsDirect& operator=(const ComputeBoundaryCellsDirect&) = delete; ComputeBoundaryCellsDirect& operator=(ComputeBoundaryCellsDirect&&) noexcept = delete; + /** + * @brief Executes the in-core boundary cell counting algorithm. + * + * Iterates every voxel in Z-Y-X order and counts how many of its 6 face + * neighbors belong to a different feature. Optionally counts volume-boundary + * faces and optionally ignores feature 0 neighbors. + * + * @return Result<> indicating success or errors. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; ///< Algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cooperative cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message handler. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp index 3176e89a2a..2d66dc6438 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.cpp @@ -9,6 +9,41 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeBoundaryCellsScanline -- Out-of-Core Algorithm +// +// This file implements the OOC-optimized variant of boundary cell counting. +// The algorithm produces the same output as ComputeBoundaryCellsDirect: for +// each voxel, an Int8 count of how many of its 6 face neighbors belong to a +// different feature. +// +// KEY DESIGN PRINCIPLE: All data access is strictly sequential by Z-slice. +// The FeatureIds input is read one Z-slice at a time using copyIntoBuffer(), +// and the BoundaryCells output is written one Z-slice at a time using +// copyFromBuffer(). This ensures that chunked/OOC storage backends serve +// data in large sequential reads rather than random single-element lookups. +// +// ROLLING WINDOW APPROACH: +// To check the -Z and +Z neighbors of a voxel at slice z, we need data from +// slices z-1 and z+1. Rather than re-reading slices, we maintain 3 buffers: +// +// prevSlice = FeatureIds for Z-slice (z-1) +// curSlice = FeatureIds for Z-slice (z) <-- being processed +// nextSlice = FeatureIds for Z-slice (z+1) +// +// After processing slice z, we rotate the buffers (via std::swap, which is +// O(1) for vectors -- just swaps internal pointers) and load slice z+2 into +// the now-free buffer. This means each Z-slice is read from disk exactly once. +// +// Within a Z-slice, the X and Y neighbor lookups use simple index arithmetic +// on curSlice: +/-1 for X neighbors, +/-dimX for Y neighbors. These are all +// in the same contiguous buffer, so there is no I/O cost. +// +// MEMORY BUDGET: 3 * (dimX * dimY * 4 bytes) for input buffers, plus +// 1 * (dimX * dimY * 1 byte) for the output buffer. For a 1024x1024 slice, +// this is ~12.6 MB total -- negligible compared to the full volume. +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeBoundaryCellsScanline::ComputeBoundaryCellsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeBoundaryCellsInputValues* inputValues) @@ -24,59 +59,102 @@ ComputeBoundaryCellsScanline::~ComputeBoundaryCellsScanline() noexcept = default // ----------------------------------------------------------------------------- /** - * @brief Counts boundary faces per voxel using Z-slice rolling window iteration. - * OOC path: reads/writes one Z-slice at a time via copyIntoBuffer/copyFromBuffer, - * keeping a 3-slice rolling window (prevSlice, curSlice, nextSlice) for Z-neighbor access. + * @brief Counts boundary faces per voxel using a 3-slice rolling window with + * sequential bulk I/O for out-of-core storage compatibility. + * + * The algorithm has three phases: + * + * **Phase 1 -- Initialization**: Allocate three input slice buffers + * (prevSlice, curSlice, nextSlice) and one output slice buffer. Load the + * first Z-slice into curSlice and (if the volume has more than one Z-slice) + * the second Z-slice into nextSlice. + * + * **Phase 2 -- Z-slice iteration**: For each Z-slice: + * - Iterate all voxels in Y-X order within curSlice. + * - For each voxel, check volume-boundary contribution (if enabled). + * - Check 6 face neighbors: + * - -Z neighbor: read from prevSlice at the same (y, x) position. + * - +Z neighbor: read from nextSlice at the same (y, x) position. + * - -Y neighbor: read from curSlice at (sliceIndex - dimX). + * - +Y neighbor: read from curSlice at (sliceIndex + dimX). + * - -X neighbor: read from curSlice at (sliceIndex - 1). + * - +X neighbor: read from curSlice at (sliceIndex + 1). + * - Write the finished output slice to BoundaryCells via copyFromBuffer(). + * - Rotate the rolling window: prevSlice <- curSlice <- nextSlice, then + * load the next-next Z-slice into the freed buffer. + * + * **Phase 3 -- Completion**: After all Z-slices are processed, the output + * array contains the boundary cell counts for the entire volume. + * + * @return Result<> indicating success or cancellation. */ Result<> ComputeBoundaryCellsScanline::operator()() { + // -- Phase 1: Initialization -- + const auto& imageGeometry = m_DataStructure.getDataRefAs(m_InputValues->ImageGeometryPath); const SizeVec3 udims = imageGeometry.getDimensions(); const int64 dimX = static_cast(udims[0]); const int64 dimY = static_cast(udims[1]); const int64 dimZ = static_cast(udims[2]); + // Access the data stores via references. Even though these may be OOC stores, + // we never use operator[] on them -- only copyIntoBuffer/copyFromBuffer. auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); auto& boundaryCellsStore = m_DataStructure.getDataAs(m_InputValues->BoundaryCellsArrayName)->getDataStoreRef(); + // See ComputeBoundaryCellsDirect for explanation of ignoreFeatureZeroVal logic. int32 ignoreFeatureZeroVal = 0; if(!m_InputValues->IgnoreFeatureZero) { ignoreFeatureZeroVal = -1; } + // Each Z-slice contains dimY * dimX voxels. This is the unit of bulk I/O. const usize sliceSize = static_cast(dimY) * static_cast(dimX); + // Allocate the 3-slice rolling window for FeatureIds input and 1 output buffer. + // Using std::vector ensures the buffers are contiguous and compatible with + // nonstd::span for the copyIntoBuffer/copyFromBuffer API. std::vector prevSlice(sliceSize); std::vector curSlice(sliceSize); std::vector nextSlice(sliceSize); std::vector outputSlice(sliceSize); - // Load first Z-slice + // Load the first Z-slice (z=0) into curSlice. The offset is 0 (start of array). featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + // If there is a second Z-slice, pre-load it into nextSlice. The offset is + // sliceSize elements (one full Z-slice into the flat array). if(dimZ > 1) { featureIdsStore.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); } + // -- Phase 2: Z-slice iteration with rolling window -- + for(int64 zIdx = 0; zIdx < dimZ; zIdx++) { + // Check for user cancellation once per Z-slice if(m_ShouldCancel) { return {}; } - // Process current slice + + // Process every voxel in the current Z-slice for(int64 yIdx = 0; yIdx < dimY; yIdx++) { const int64 rowOffset = yIdx * dimX; for(int64 xIdx = 0; xIdx < dimX; xIdx++) { + // sliceIndex is the flat index within the current Z-slice buffer. + // This is NOT the global voxel index -- it ranges from 0 to sliceSize-1. const int64 sliceIndex = rowOffset + xIdx; int8 onSurf = 0; const int32 feature = curSlice[sliceIndex]; if(feature >= 0) { + // Volume boundary contribution -- same logic as the Direct variant if(m_InputValues->IncludeVolumeBoundary) { if(dimX > 2 && (xIdx == 0 || xIdx == dimX - 1)) @@ -98,7 +176,9 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // -Z neighbor + // -Z neighbor: This voxel's -Z neighbor is the same (y, x) position + // in the PREVIOUS Z-slice. Because prevSlice is an in-memory buffer, + // this lookup is a simple array index -- no disk I/O. if(zIdx > 0) { if(prevSlice[sliceIndex] != feature && prevSlice[sliceIndex] > ignoreFeatureZeroVal) @@ -107,7 +187,8 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // -Y neighbor + // -Y neighbor: One row back in the current slice. The offset is + // -dimX because each row has dimX elements. if(yIdx > 0) { const int64 neighborIdx = sliceIndex - dimX; @@ -117,7 +198,7 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // -X neighbor + // -X neighbor: One column back in the current row. if(xIdx > 0) { const int64 neighborIdx = sliceIndex - 1; @@ -127,7 +208,7 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // +X neighbor + // +X neighbor: One column forward in the current row. if(xIdx < dimX - 1) { const int64 neighborIdx = sliceIndex + 1; @@ -137,7 +218,7 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // +Y neighbor + // +Y neighbor: One row forward in the current slice. if(yIdx < dimY - 1) { const int64 neighborIdx = sliceIndex + dimX; @@ -147,7 +228,8 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // +Z neighbor + // +Z neighbor: This voxel's +Z neighbor is the same (y, x) position + // in the NEXT Z-slice buffer -- again, a simple in-memory lookup. if(zIdx < dimZ - 1) { if(nextSlice[sliceIndex] != feature && nextSlice[sliceIndex] > ignoreFeatureZeroVal) @@ -161,17 +243,28 @@ Result<> ComputeBoundaryCellsScanline::operator()() } } - // Write output slice + // Write the completed output Z-slice to the BoundaryCells array. + // The offset is zIdx * sliceSize elements into the flat output array. boundaryCellsStore.copyFromBuffer(static_cast(zIdx) * sliceSize, nonstd::span(outputSlice.data(), sliceSize)); - // Shift rolling window + // Rotate the rolling window for the next iteration: + // prevSlice gets the old curSlice data (now z-1 relative to the next iteration) + // curSlice gets the old nextSlice data (now z relative to the next iteration) + // nextSlice buffer is now free to receive new data + // std::swap on vectors is O(1) -- it just swaps internal pointers, not data. std::swap(prevSlice, curSlice); std::swap(curSlice, nextSlice); + + // Load the next-next Z-slice (z+2) into the freed nextSlice buffer. + // This is the only disk read per iteration -- one sequential bulk read. if(zIdx + 2 < dimZ) { featureIdsStore.copyIntoBuffer(static_cast(zIdx + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); } } + // -- Phase 3: Done -- + // All Z-slices have been processed and written. The BoundaryCells array now + // contains the boundary face count for every voxel in the volume. return {}; } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp index 0628f62a5f..ada7210eeb 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundaryCellsScanline.hpp @@ -11,14 +11,55 @@ struct ComputeBoundaryCellsInputValues; /** * @class ComputeBoundaryCellsScanline - * @brief Out-of-core algorithm for ComputeBoundaryCells. Iterates Z-slices sequentially - * using copyIntoBuffer/copyFromBuffer bulk I/O with a 3-slice rolling window - * (prevSlice, curSlice, nextSlice) for Z-neighbor access. Selected by DispatchAlgorithm - * when any input array is backed by ZarrStore. + * @brief Out-of-core (OOC) optimized algorithm for counting boundary cells using a + * Z-slice rolling window with sequential bulk I/O. + * + * **The problem this solves**: When the FeatureIds array is stored out-of-core in + * chunked format (e.g., Zarr/HDF5 chunks on disk), each call to operator[] may + * trigger a disk read to load the chunk containing that element. The boundary-cell + * algorithm needs to access each voxel AND its 6 face neighbors. The +/-Z neighbors + * are dimX*dimY elements away in the flat index space, which almost certainly live + * in a different chunk. This means up to 3 chunk loads per voxel (current, prev-Z, + * next-Z chunks), causing catastrophic "chunk thrashing" that makes the algorithm + * 100-1000x slower than in-core execution. + * + * **How the rolling window solves it**: Instead of random operator[] access, this + * algorithm reads one complete Z-slice at a time using copyIntoBuffer(), which + * performs a single sequential bulk read per slice. Three std::vector buffers + * hold the previous, current, and next Z-slices simultaneously: + * + * - prevSlice: Z-slice at (z-1) -- needed for -Z neighbor lookups + * - curSlice: Z-slice at (z) -- the slice being processed + * - nextSlice: Z-slice at (z+1) -- needed for +Z neighbor lookups + * + * Within a single Z-slice, all X and Y neighbor lookups are simple index arithmetic + * on the curSlice buffer (+/-1 for X, +/-dimX for Y). After processing curSlice, + * the window shifts: prevSlice <- curSlice, curSlice <- nextSlice, and a new + * nextSlice is loaded from disk. The output is similarly written one Z-slice at a + * time using copyFromBuffer(). + * + * This guarantees that the entire algorithm reads and writes the FeatureIds and + * BoundaryCells arrays in strictly sequential order, with exactly one bulk I/O + * operation per Z-slice -- optimal for chunked storage. + * + * **Memory overhead**: 3 input buffers (prevSlice, curSlice, nextSlice) each of + * size dimX * dimY * sizeof(int32), plus 1 output buffer of size dimX * dimY * + * sizeof(int8). For a 1000x1000 slice, this is approximately 12 MB total. + * + * @see ComputeBoundaryCellsDirect for the in-core variant. + * @see ComputeBoundaryCells for the dispatcher. + * @see DispatchAlgorithm for the selection mechanism. */ class SIMPLNXCORE_EXPORT ComputeBoundaryCellsScanline { public: + /** + * @brief Constructs the OOC-optimized boundary cell counter. + * @param dataStructure The DataStructure containing FeatureIds and BoundaryCells arrays. + * @param mesgHandler Handler for progress/info messages. + * @param shouldCancel Atomic flag for cooperative cancellation. + * @param inputValues Algorithm parameters (geometry path, array paths, flags). + */ ComputeBoundaryCellsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeBoundaryCellsInputValues* inputValues); ~ComputeBoundaryCellsScanline() noexcept; @@ -27,13 +68,18 @@ class SIMPLNXCORE_EXPORT ComputeBoundaryCellsScanline ComputeBoundaryCellsScanline& operator=(const ComputeBoundaryCellsScanline&) = delete; ComputeBoundaryCellsScanline& operator=(ComputeBoundaryCellsScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized boundary cell counting algorithm using + * a 3-slice rolling window with copyIntoBuffer/copyFromBuffer bulk I/O. + * @return Result<> indicating success or errors. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeBoundaryCellsInputValues* m_InputValues = nullptr; ///< Algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cooperative cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message handler. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp index ebc29363ff..09bd7c8a77 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.cpp @@ -236,6 +236,27 @@ ComputeEuclideanDistMap::ComputeEuclideanDistMap(DataStructure& dataStructure, c // ----------------------------------------------------------------------------- ComputeEuclideanDistMap::~ComputeEuclideanDistMap() noexcept = default; +// ----------------------------------------------------------------------------- +/** + * @brief Core distance map computation, templated on the output type (int32 for + * Manhattan distance, float32 for Euclidean distance). + * + * OOC optimization strategy: + * The original implementation accessed FeatureIds and distance DataStores through + * per-element virtual dispatch in three passes: boundary identification, iterative + * Manhattan propagation, and Euclidean distance correction. Each pass iterated + * over all voxels, causing O(totalVoxels * passes) chunk operations for OOC data. + * + * The optimized implementation front-loads all DataStore I/O: + * 1. Bulk-read the entire FeatureIds array into featureIdsBuf. + * 2. Fill distance stores with -1, then bulk-read into local buffers (gbDistBuf, etc.). + * 3. Boundary identification runs entirely on local buffers. + * 4. Each ComputeDistanceMapImpl worker receives raw pointers (not DataStore refs), + * so propagation and distance correction use plain memory access. + * 5. Workers write results back via a single copyFromBuffer() at the end. + * + * This reduces OOC round-trips from O(totalVoxels * passes) to O(1) per map. + */ // ----------------------------------------------------------------------------- template void FindDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMapInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& messageHandler) @@ -246,7 +267,9 @@ void FindDistanceMap(DataStructure& dataStructure, const ComputeEuclideanDistMap const auto& featureIdsStoreRef = dataStructure.getDataRefAs(inputValues->FeatureIdsArrayPath).getDataStoreRef(); usize totalVoxels = featureIdsStoreRef.getNumberOfTuples(); - // Bulk-read featureIds into a local buffer to avoid per-element OOC access + // Bulk-read the entire FeatureIds array into a local buffer. This is a + // full-volume read but is done once, vs. the original per-element access + // that triggered a chunk operation per voxel per pass. std::vector featureIdsBuf(totalVoxels); featureIdsStoreRef.copyIntoBuffer(0, nonstd::span(featureIdsBuf.data(), totalVoxels)); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.hpp index e3c2e27dbf..8dd4343af9 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeEuclideanDistMap.hpp @@ -9,21 +9,49 @@ namespace nx::core { +/** + * @struct ComputeEuclideanDistMapInputValues + * @brief Holds all user-configured parameters for the ComputeEuclideanDistMap algorithm. + */ struct SIMPLNXCORE_EXPORT ComputeEuclideanDistMapInputValues { - bool CalcManhattanDist; - bool DoBoundaries; - bool DoTripleLines; - bool DoQuadPoints; - DataPath FeatureIdsArrayPath; - DataPath GBDistancesArrayPath; - DataPath TJDistancesArrayPath; - DataPath QPDistancesArrayPath; - DataPath InputImageGeometry; + bool CalcManhattanDist; ///< If true, output Manhattan (city-block) distances as int32; otherwise Euclidean as float32. + bool DoBoundaries; ///< Compute distance to nearest grain boundary (2+ unique neighbors). + bool DoTripleLines; ///< Compute distance to nearest triple line (3+ unique neighbors). + bool DoQuadPoints; ///< Compute distance to nearest quadruple point (4+ unique neighbors). + DataPath FeatureIdsArrayPath; ///< Per-cell Feature ID array (int32). + DataPath GBDistancesArrayPath; ///< Output: grain boundary distance map. + DataPath TJDistancesArrayPath; ///< Output: triple junction distance map. + DataPath QPDistancesArrayPath; ///< Output: quadruple point distance map. + DataPath InputImageGeometry; ///< Path to the ImageGeom. }; /** - * @class + * @class ComputeEuclideanDistMap + * @brief Computes distance maps from each voxel to the nearest grain boundary, + * triple junction, and/or quadruple point using iterative neighbor propagation + * followed by optional Euclidean distance correction. + * + * The algorithm first identifies boundary voxels by checking each voxel's 6 face + * neighbors for different Feature IDs. Voxels with 2+ distinct neighbors are grain + * boundaries, 3+ are triple lines, 4+ are quadruple points. It then propagates + * distances outward using iterative "city-block" expansion and optionally converts + * to true Euclidean distances. + * + * @section ooc_optimization Out-of-Core Optimization + * The original implementation accessed FeatureIds and distance DataStores through + * per-element virtual dispatch in multiple passes (boundary identification, iterative + * propagation, final distance write-back). For OOC data, this caused severe chunk + * thrashing across all passes. + * + * The optimized implementation: + * 1. Bulk-reads the entire FeatureIds array into a local std::vector via + * copyIntoBuffer() at the start of FindDistanceMap(). + * 2. Bulk-reads each distance store into local buffers after the initial fill(-1). + * 3. All boundary identification and distance propagation operates on local buffers. + * 4. Each ComputeDistanceMapImpl worker receives raw pointers to the pre-loaded + * buffers instead of DataStore references, eliminating virtual dispatch. + * 5. Results are written back via a single copyFromBuffer() call per map. */ class SIMPLNXCORE_EXPORT ComputeEuclideanDistMap { @@ -38,22 +66,34 @@ class SIMPLNXCORE_EXPORT ComputeEuclideanDistMap using EnumType = uint32_t; + /** + * @enum MapType + * @brief Identifies which type of distance map a ComputeDistanceMapImpl instance computes. + */ enum class MapType : EnumType { - FeatureBoundary = 0, //!< - TripleJunction = 1, //!< - QuadPoint = 2, //!< + FeatureBoundary = 0, ///< Distance to nearest grain boundary (2+ unique neighbors). + TripleJunction = 1, ///< Distance to nearest triple junction (3+ unique neighbors). + QuadPoint = 2, ///< Distance to nearest quadruple point (4+ unique neighbors). }; + /** + * @brief Executes the distance map computation, dispatching int32 (Manhattan) or float32 (Euclidean). + * @return Result<> indicating success or error. + */ Result<> operator()(); + /** + * @brief Returns the cancellation flag reference. + * @return const reference to the atomic cancellation boolean. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeEuclideanDistMapInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const ComputeEuclideanDistMapInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp index f742f172cd..c2e274d8ea 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.cpp @@ -16,6 +16,9 @@ using namespace nx::core; namespace { +/// Number of FeatureId tuples to read per bulk I/O call. 64K tuples balances +/// between minimizing the number of copyIntoBuffer() round-trips and keeping +/// the per-chunk buffer small enough to stay in L2 cache. constexpr usize k_ChunkTuples = 65536; } // namespace @@ -38,6 +41,29 @@ const std::atomic_bool& ComputeFeatureCentroids::getCancel() return m_ShouldCancel; } +// ----------------------------------------------------------------------------- +/** + * @brief Computes the centroid of each feature using chunked bulk I/O and Kahan + * summation. + * + * Algorithm: + * 1. Read FeatureIds in 64K-tuple chunks via copyIntoBuffer(). + * 2. For each voxel in the chunk, compute its XYZ voxel-center coordinate from + * the flat index, origin, and spacing (avoiding ImageGeom::getCoords() virtual call). + * 3. Accumulate each coordinate component into a Kahan sum keyed by feature ID. + * 4. Track per-feature min/max XYZ indices for periodic boundary detection. + * 5. Divide accumulated sums by voxel counts to produce centroids. + * 6. Write all centroids to the output DataStore in one copyFromBuffer() call. + * 7. If IsPeriodic, adjust centroids for features that wrap around geometry bounds. + * + * OOC optimization rationale: + * The previous implementation used a ParallelDataAlgorithm with per-element + * operator[] access on FeatureIds, plus DataStore-backed accumulation arrays. + * For OOC FeatureIds stores, every operator[] triggered a chunk load. The + * chunked approach reduces chunk operations from O(totalVoxels) to + * O(totalVoxels / 64K), and plain-vector accumulators eliminate virtual dispatch + * entirely for the feature-level bookkeeping. + */ // ----------------------------------------------------------------------------- Result<> ComputeFeatureCentroids::operator()() { @@ -58,8 +84,10 @@ Result<> ComputeFeatureCentroids::operator()() const usize yPoints = imageGeom.getNumYCells(); const usize zPoints = imageGeom.getNumZCells(); - // Plain vectors for accumulation (feature-level, small) to avoid - // AbstractDataStore virtual dispatch in the hot loop. + // Plain std::vectors for accumulation instead of DataStore-backed arrays. + // These are feature-level (small, typically thousands of elements), so they + // fit easily in memory. Using plain vectors avoids AbstractDataStore virtual + // dispatch on every accumulation step in the hot voxel loop. const usize featureElems3 = totalFeatures * 3; const usize featureElems2 = totalFeatures * 2; std::vector kahanSum(featureElems3, 0.0); @@ -81,6 +109,8 @@ Result<> ComputeFeatureCentroids::operator()() const usize totalVoxels = xPoints * yPoints * zPoints; const usize xySize = xPoints * yPoints; + // Main voxel loop: read FeatureIds in 64K-tuple chunks via copyIntoBuffer(). + // Each chunk is processed from the local buffer with zero OOC overhead. auto featureIdBuf = std::make_unique(k_ChunkTuples); for(usize offset = 0; offset < totalVoxels; offset += k_ChunkTuples) { @@ -130,6 +160,8 @@ Result<> ComputeFeatureCentroids::operator()() } } + // Finalize centroids: divide Kahan sums by voxel counts, then bulk-write + // all centroids to the output DataStore in a single copyFromBuffer() call. std::vector centroidsBuf(featureElems3, 0.0f); for(usize featureId = 0; featureId < totalFeatures; featureId++) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.hpp index 2a4a50bde5..bf6cc617ed 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureCentroids.hpp @@ -9,17 +9,37 @@ namespace nx::core { +/** + * @struct ComputeFeatureCentroidsInputValues + * @brief Holds all user-configured parameters for the ComputeFeatureCentroids algorithm. + */ struct SIMPLNXCORE_EXPORT ComputeFeatureCentroidsInputValues { - DataPath FeatureIdsArrayPath; - DataPath CentroidsArrayPath; - DataPath ImageGeometryPath; - DataPath FeatureAttributeMatrixPath; - bool IsPeriodic = false; + DataPath FeatureIdsArrayPath; ///< Path to the per-cell Feature ID array (int32). + DataPath CentroidsArrayPath; ///< Output: per-feature centroid array (float32, 3-component). + DataPath ImageGeometryPath; ///< Path to the ImageGeom providing voxel coordinates. + DataPath FeatureAttributeMatrixPath; ///< Path to the Feature-level Attribute Matrix. + bool IsPeriodic = false; ///< If true, adjust centroids for features wrapping around periodic boundaries. }; /** - * @class + * @class ComputeFeatureCentroids + * @brief Computes the centroid (average XYZ position) of each feature in an + * ImageGeom by iterating over all voxels and accumulating coordinates using + * Kahan summation for numerical stability. + * + * @section ooc_optimization Out-of-Core Optimization + * The original implementation used a ParallelDataAlgorithm that accessed the + * FeatureIds array element-by-element through AbstractDataStore virtual dispatch. + * For OOC data this caused a chunk load/evict cycle per voxel. + * + * The optimized implementation reads the FeatureIds array in fixed-size chunks + * (64K tuples) via copyIntoBuffer() into a stack-allocated buffer, then processes + * each chunk purely from local memory. All accumulation arrays (Kahan sums, + * compensators, voxel counts, XYZ ranges) are plain std::vectors rather than + * DataStore-backed arrays, eliminating virtual dispatch in the hot loop. The + * final centroids are written back to the output DataStore in a single + * copyFromBuffer() call. */ class SIMPLNXCORE_EXPORT ComputeFeatureCentroids { @@ -32,15 +52,23 @@ class SIMPLNXCORE_EXPORT ComputeFeatureCentroids ComputeFeatureCentroids& operator=(const ComputeFeatureCentroids&) = delete; ComputeFeatureCentroids& operator=(ComputeFeatureCentroids&&) noexcept = delete; + /** + * @brief Executes the centroid computation over all voxels using chunked bulk I/O. + * @return Result<> indicating success or error. + */ Result<> operator()(); + /** + * @brief Returns the cancellation flag reference. + * @return const reference to the atomic cancellation boolean. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeFeatureCentroidsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const ComputeFeatureCentroidsInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp index 13c6a1d086..bc9918a411 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.cpp @@ -127,6 +127,18 @@ const std::atomic_bool& ComputeFeatureClustering::getCancel() return m_ShouldCancel; } +// ----------------------------------------------------------------------------- +/** + * @brief Computes the radial distribution function (RDF) for features in a + * specified phase. The O(n^2) pairwise distance computation is the bottleneck. + * + * OOC optimization: The FeaturePhases and Centroids arrays are accessed in the + * inner O(n^2) loop. For OOC data, per-element virtual dispatch inside a + * quadratic loop causes n^2 chunk operations -- catastrophic for performance. + * Both arrays are bulk-read into local std::vectors at the start via + * copyIntoBuffer(). The RDF histogram is also accumulated into a local vector + * and written back via copyFromBuffer() after normalization. + */ // ----------------------------------------------------------------------------- Result<> ComputeFeatureClustering::operator()() { @@ -134,7 +146,8 @@ Result<> ComputeFeatureClustering::operator()() const auto& featurePhasesStoreRef = m_DataStructure.getDataAs(m_InputValues->FeaturePhasesArrayPath)->getDataStoreRef(); const auto& centroidsStoreRef = m_DataStructure.getDataAs(m_InputValues->CentroidsArrayPath)->getDataStoreRef(); - // Cache feature-level arrays locally to avoid per-element OOC overhead in O(n^2) loops + // Bulk-read feature-level arrays into local caches. These are small (one entry + // per feature, typically thousands) but accessed O(n^2) times in the distance loop. const usize numPhases = featurePhasesStoreRef.getSize(); std::vector featurePhasesCache(numPhases); featurePhasesStoreRef.copyIntoBuffer(0, nonstd::span(featurePhasesCache.data(), numPhases)); @@ -147,7 +160,9 @@ Result<> ComputeFeatureClustering::operator()() auto& rdfStore = m_DataStructure.getDataAs(m_InputValues->RDFArrayName)->getDataStoreRef(); auto& minMaxDistancesStore = m_DataStructure.getDataAs(m_InputValues->MaxMinArrayName)->getDataStoreRef(); - // Cache rdf output array locally + // Accumulate RDF bins into a local vector to avoid per-increment OOC overhead. + // The original code used rdfStore[index].inc() which triggers a DataStore + // virtual call per bin increment. const usize rdfSize = rdfStore.getSize(); std::vector rdfCache(rdfSize, 0.0f); std::unique_ptr maskCompare; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.hpp index 791ef7f6d5..a89edb7551 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureClustering.hpp @@ -10,27 +10,46 @@ namespace nx::core { +/** + * @struct ComputeFeatureClusteringInputValues + * @brief Holds all user-configured parameters for the ComputeFeatureClustering algorithm. + */ struct SIMPLNXCORE_EXPORT ComputeFeatureClusteringInputValues { - DataPath ImageGeometryPath; - int32 NumberOfBins; - int32 PhaseNumber; - bool RemoveBiasedFeatures; - uint64 SeedValue; - DataPath FeaturePhasesArrayPath; - DataPath CentroidsArrayPath; - DataPath BiasedFeaturesArrayPath; - DataPath CellEnsembleAttributeMatrixName; - DataPath ClusteringListArrayName; - DataPath RDFArrayName; - DataPath MaxMinArrayName; + DataPath ImageGeometryPath; ///< Path to the ImageGeom providing box dimensions. + int32 NumberOfBins; ///< Number of histogram bins for the RDF. + int32 PhaseNumber; ///< Ensemble/phase to compute clustering for. + bool RemoveBiasedFeatures; ///< If true, exclude features flagged as biased. + uint64 SeedValue; ///< Random seed for the reference random distribution. + DataPath FeaturePhasesArrayPath; ///< Per-feature phase/ensemble ID array. + DataPath CentroidsArrayPath; ///< Per-feature centroid array (float32, 3-component). + DataPath BiasedFeaturesArrayPath; ///< Per-feature bias flag array (used when RemoveBiasedFeatures is true). + DataPath CellEnsembleAttributeMatrixName; ///< Ensemble-level Attribute Matrix. + DataPath ClusteringListArrayName; ///< Output: NeighborList of inter-feature distances. + DataPath RDFArrayName; ///< Output: RDF histogram array (float32). + DataPath MaxMinArrayName; ///< Output: min/max separation distances (float32, 2-component). }; /** * @class ComputeFeatureClustering - * @brief This filter determines the radial distribution function (RDF), as a histogram, of a given set of Features. + * @brief Computes the radial distribution function (RDF) for features of a + * specified phase by measuring all pairwise inter-centroid distances and + * normalizing against a random reference distribution. + * + * The algorithm has O(n^2) complexity in the number of features of the target + * phase, since it computes all pairwise distances. The RDF is binned into a + * user-specified number of equal-width bins spanning the minimum to maximum + * inter-feature distance, then normalized by a Monte Carlo random distribution. + * + * @section ooc_optimization Out-of-Core Optimization + * The feature-level arrays (FeaturePhases, Centroids, RDF) are accessed in the + * inner O(n^2) loop. For OOC data, per-element virtual dispatch on every access + * inside a quadratic loop is prohibitively expensive. The optimized implementation + * bulk-reads the entire FeaturePhases and Centroids arrays into local std::vectors + * via copyIntoBuffer() at the start, and accumulates RDF bins into a local vector. + * The final RDF is written back to the output DataStore in a single copyFromBuffer() + * call after normalization. */ - class SIMPLNXCORE_EXPORT ComputeFeatureClustering { public: @@ -42,15 +61,23 @@ class SIMPLNXCORE_EXPORT ComputeFeatureClustering ComputeFeatureClustering& operator=(const ComputeFeatureClustering&) = delete; ComputeFeatureClustering& operator=(ComputeFeatureClustering&&) noexcept = delete; + /** + * @brief Executes the RDF clustering computation. + * @return Result<> indicating success or error. + */ Result<> operator()(); + /** + * @brief Returns the cancellation flag reference. + * @return const reference to the atomic cancellation boolean. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeFeatureClusteringInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const ComputeFeatureClusteringInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp index f571f25996..10a6448600 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.cpp @@ -8,6 +8,18 @@ using namespace nx::core; +// ============================================================================= +// ComputeFeatureNeighbors — Dispatcher +// +// This file contains only the dispatch logic. The actual algorithm implementations +// live in ComputeFeatureNeighborsDirect.cpp (in-core) and +// ComputeFeatureNeighborsScanline.cpp (out-of-core). +// +// The dispatch checks the FeatureIds array's storage type: if it uses chunked +// on-disk storage (OOC), the Scanline variant is selected to avoid chunk thrashing. +// Otherwise, the Direct variant is used for optimal in-memory performance. +// ============================================================================= + // ----------------------------------------------------------------------------- ComputeFeatureNeighbors::ComputeFeatureNeighbors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureNeighborsInputValues* inputValues) @@ -22,8 +34,19 @@ ComputeFeatureNeighbors::ComputeFeatureNeighbors(DataStructure& dataStructure, c ComputeFeatureNeighbors::~ComputeFeatureNeighbors() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to the appropriate algorithm variant based on storage type. + * + * Uses DispatchAlgorithm() to check whether the FeatureIds array + * is backed by out-of-core (chunked) storage. If so, the Scanline variant is used; + * otherwise, the Direct variant is selected. + * + * Both variants receive identical constructor arguments and produce identical output. + */ Result<> ComputeFeatureNeighbors::operator()() { + // Check the FeatureIds array — this is the primary input array that both + // variants iterate over, so its storage type determines which path to take. auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath); return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp index 810e3a4dd5..3da93cedf1 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighbors.hpp @@ -14,28 +14,66 @@ namespace nx::core { +/** + * @struct ComputeFeatureNeighborsInputValues + * @brief Input parameter bundle for the ComputeFeatureNeighbors algorithm. + * + * Aggregates all DataPaths and boolean flags needed by both the in-core (Direct) + * and out-of-core (Scanline) variants of the feature neighbor computation. + */ struct SIMPLNXCORE_EXPORT ComputeFeatureNeighborsInputValues { - DataPath BoundaryCellsPath; - AttributeMatrixSelectionParameter::ValueType CellFeatureArrayPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - DataPath NeighborListPath; - DataPath NumberOfNeighborsPath; - DataPath SharedSurfaceAreaListPath; - BoolParameter::ValueType StoreBoundaryCells; - BoolParameter::ValueType StoreSurfaceFeatures; - DataPath SurfaceFeaturesPath; + DataPath BoundaryCellsPath; ///< Output Int8 array marking how many different-feature face neighbors each cell has + AttributeMatrixSelectionParameter::ValueType CellFeatureArrayPath; ///< Attribute matrix where per-feature output arrays reside + ArraySelectionParameter::ValueType FeatureIdsPath; ///< Input Int32 array of per-cell feature IDs + GeometrySelectionParameter::ValueType InputImageGeometryPath; ///< Input ImageGeom providing dimensions and spacing + DataPath NeighborListPath; ///< Output Int32 NeighborList storing each feature's neighbor IDs + DataPath NumberOfNeighborsPath; ///< Output Int32 array storing the count of neighbors per feature + DataPath SharedSurfaceAreaListPath; ///< Output Float32 NeighborList storing shared surface area per neighbor pair + BoolParameter::ValueType StoreBoundaryCells; ///< Whether to compute and store the BoundaryCells array + BoolParameter::ValueType StoreSurfaceFeatures; ///< Whether to compute and store the SurfaceFeatures array + DataPath SurfaceFeaturesPath; ///< Output Bool array marking features that touch the geometry boundary }; /** * @class ComputeFeatureNeighbors - * @brief This algorithm implements support code for the ComputeFeatureNeighborsFilter + * @brief Dispatcher algorithm for computing feature neighbor lists and shared surface areas + * on an ImageGeom. + * + * This class acts as a thin dispatcher that selects between two concrete algorithm + * implementations at runtime: + * + * - **ComputeFeatureNeighborsDirect** (in-core): Uses per-element getValue() access with + * compile-time dimension specialization. Optimal when all arrays reside in memory. + * + * - **ComputeFeatureNeighborsScanline** (out-of-core / OOC): Uses a Z-slice rolling + * window with bulk copyIntoBuffer()/copyFromBuffer() I/O. Avoids random disk access + * when arrays are backed by chunked on-disk storage (e.g., Zarr/HDF5 chunks). + * + * The dispatch decision is made by DispatchAlgorithm() in + * AlgorithmDispatch.hpp, which checks whether any input IDataArray uses OOC storage. + * + * **Why two variants exist**: When data is stored out-of-core in compressed disk chunks, + * each random-access getValue() call may trigger a chunk load from disk, use one value, + * then evict the chunk. For a 3D image with millions of voxels, this "chunk thrashing" + * makes the algorithm 100-1000x slower. The Scanline variant reads entire Z-slices + * sequentially, keeping a rolling window of 2-3 slices in memory so that all 6 face + * neighbors can be resolved from in-memory buffers. + * + * @see ComputeFeatureNeighborsDirect + * @see ComputeFeatureNeighborsScanline + * @see AlgorithmDispatch.hpp */ - class SIMPLNXCORE_EXPORT ComputeFeatureNeighbors { public: + /** + * @brief Constructs the dispatcher with all resources needed by either algorithm variant. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ ComputeFeatureNeighbors(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeFeatureNeighborsInputValues* inputValues); ~ComputeFeatureNeighbors() noexcept; @@ -44,13 +82,17 @@ class SIMPLNXCORE_EXPORT ComputeFeatureNeighbors ComputeFeatureNeighbors& operator=(const ComputeFeatureNeighbors&) = delete; ComputeFeatureNeighbors& operator=(ComputeFeatureNeighbors&&) noexcept = delete; + /** + * @brief Dispatches to the Direct or Scanline algorithm based on storage type. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp index d67cb604a2..47ac72dea3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.cpp @@ -9,6 +9,33 @@ using namespace nx::core; +// ============================================================================= +// ComputeFeatureNeighborsDirect — In-Core Algorithm +// +// This file implements the in-core (Direct) variant of ComputeFeatureNeighbors. +// It is selected by DispatchAlgorithm when all input arrays reside in memory. +// +// ALGORITHM OVERVIEW: +// For each voxel in an ImageGeom, compare its FeatureId against the FeatureIds +// of its 6 face neighbors (+/-X, +/-Y, +/-Z). When two adjacent voxels belong +// to different features, accumulate the shared face's surface area into a +// per-feature-pair map. After all voxels are processed, convert the maps into +// NeighborList and SharedSurfaceAreaList arrays. +// +// KEY DESIGN DECISIONS: +// 1. Compile-time dimension specialization via ImageDimensionState<> templates +// eliminates runtime branching for degenerate dimensions (1D, 2D geometries). +// 2. Two-stage processing separates boundary cells (which need validity checks) +// from internal cells (where all 6 neighbors are guaranteed valid), removing +// a branch from the innermost loop of Stage 2. +// 3. Per-face surface areas use precomputed values from computeFaceSurfaceAreas() +// rather than a uniform area, fixing a DREAM3D 6.5 bug. +// +// DATA ACCESS PATTERN: +// Uses getValue() for per-element random access. This is optimal for in-memory +// DataStore where getValue() is essentially a pointer dereference. +// ============================================================================= + namespace { // ============================================================================= @@ -574,12 +601,18 @@ ComputeFeatureNeighborsDirect::~ComputeFeatureNeighborsDirect() noexcept = defau // ----------------------------------------------------------------------------- /** - * @brief In-core implementation of ComputeFeatureNeighbors using Nathan Young's - * rewritten algorithm with compile-time dimension specialization and per-face - * surface area accumulation. + * @brief In-core implementation of ComputeFeatureNeighbors. + * + * Uses Nathan Young's rewritten algorithm with compile-time dimension specialization + * and per-face surface area accumulation. + * + * Accesses FeatureIds via getValue() (per-element random access), which is optimal + * for in-memory DataStore where it is essentially a pointer dereference. For OOC + * data, the Scanline variant reads Z-slices via bulk I/O instead. * - * Uses getValue() for per-element array access (in-core optimal). - * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * The function dispatches to one of 4 template specializations based on boolean + * combinations of StoreSurfaceFeatures and StoreBoundaryCells, eliminating those + * branches from the innermost loop via constexpr if. */ Result<> ComputeFeatureNeighborsDirect::operator()() { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp index f6fdfd131f..916e41e0a3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsDirect.hpp @@ -30,6 +30,13 @@ struct ComputeFeatureNeighborsInputValues; class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsDirect { public: + /** + * @brief Constructs the in-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ ComputeFeatureNeighborsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeFeatureNeighborsInputValues* inputValues); ~ComputeFeatureNeighborsDirect() noexcept; @@ -38,13 +45,25 @@ class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsDirect ComputeFeatureNeighborsDirect& operator=(const ComputeFeatureNeighborsDirect&) = delete; ComputeFeatureNeighborsDirect& operator=(ComputeFeatureNeighborsDirect&&) noexcept = delete; + /** + * @brief Executes the in-core feature neighbor computation. + * + * Uses Nathan Young's two-stage algorithm with compile-time dimension specialization: + * - Stage 1: Process boundary cells (corners, edges, faces) with validity checks + * - Stage 2: Process internal cells with all 6 neighbors guaranteed valid + * + * Per-face surface areas are computed using precomputed face dimensions rather + * than a uniform area, fixing a bug from DREAM3D 6.5. + * + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp index 4c801b8b21..4bedd0dcd3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.cpp @@ -11,6 +11,49 @@ using namespace nx::core; +// ============================================================================= +// ComputeFeatureNeighborsScanline — Out-of-Core (OOC) Algorithm +// +// This file implements the out-of-core (Scanline) variant of ComputeFeatureNeighbors. +// It is selected by DispatchAlgorithm when any input array uses chunked on-disk +// storage (e.g., ZarrStore / HDF5 chunked store). +// +// PROBLEM: +// The Direct variant uses per-element getValue() calls. When data is stored +// out-of-core in compressed chunks on disk, each getValue() may trigger: +// 1. Decompress and load the containing chunk from disk +// 2. Return the single requested value +// 3. Evict the chunk when the LRU cache fills +// For a 3D image with millions of voxels, this "chunk thrashing" makes the +// algorithm catastrophically slow (100-1000x slower than in-core). +// +// SOLUTION — Z-SLICE ROLLING WINDOW: +// Instead of random per-element access, read entire Z-slices sequentially +// using copyIntoBuffer() bulk I/O. Maintain a rolling window of 3 slices: +// - prevSlice: Z-slice at (z-1), needed for -Z neighbor lookups +// - curSlice: Z-slice at (z), the slice currently being processed +// - nextSlice: Z-slice at (z+1), needed for +Z neighbor lookups +// +// Within each slice, +/-X and +/-Y neighbors are resolved by simple index +// arithmetic on the curSlice buffer. After processing a slice, the buffers +// rotate: prev <- cur, cur <- next, and the next slice is loaded from disk. +// +// MEMORY BUDGET: +// 3 slices of FeatureIds (int32) + 1 slice of BoundaryCells (int8) +// = 3 * (dimX * dimY * 4 bytes) + (dimX * dimY * 1 byte) +// For a 500x500 slice, this is ~3.25 MB regardless of how many Z-slices exist. +// +// OUTPUT WRITING: +// BoundaryCells output is also written one Z-slice at a time via copyFromBuffer(), +// maintaining the sequential I/O pattern for the output store as well. +// +// FEATURE ID VALIDATION: +// Unlike the Direct variant which scans all FeatureIds up front to find the +// maximum, the Scanline variant tracks the maximum FeatureId during the main +// loop to avoid a separate full-volume OOC scan. The `feature < totalFeatures` +// guard prevents out-of-bounds access during processing. +// ============================================================================= + // ----------------------------------------------------------------------------- ComputeFeatureNeighborsScanline::ComputeFeatureNeighborsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeFeatureNeighborsInputValues* inputValues) @@ -80,7 +123,11 @@ Result<> ComputeFeatureNeighborsScanline::operator()() // `feature < totalFeatures` guard prevents out-of-bounds access. int32 observedMaxFeatureId = 0; - // 3-slice rolling window for Z-sequential bulk I/O + // 3-slice rolling window for Z-sequential bulk I/O. + // Each buffer holds one complete Z-slice of FeatureIds (dimX * dimY int32 values). + // At any point during processing, prevSlice holds z-1, curSlice holds z, and + // nextSlice holds z+1. After processing slice z, the buffers rotate via std::swap + // so that no data is copied — only the pointers change. std::vector prevSlice(sliceSize); std::vector curSlice(sliceSize); std::vector nextSlice(sliceSize); @@ -235,11 +282,14 @@ Result<> ComputeFeatureNeighborsScanline::operator()() boundaryCellsStore->copyFromBuffer(static_cast(z) * sliceSize, nonstd::span(boundaryCellsSlice.data(), sliceSize)); } - // Rotate the rolling window + // Rotate the rolling window: prev <- cur <- next, then load z+2 into next. + // std::swap only exchanges internal pointers/size, not element data, so this + // is O(1) regardless of slice size. std::swap(prevSlice, curSlice); std::swap(curSlice, nextSlice); if(z + 2 < dimZ) { + // Load the next slice ahead of time so it is ready when we advance to z+1. featureIds.copyIntoBuffer(static_cast(z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); } } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp index 1cc17c46c3..e680a92e57 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureNeighborsScanline.hpp @@ -30,6 +30,13 @@ struct ComputeFeatureNeighborsInputValues; class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsScanline { public: + /** + * @brief Constructs the out-of-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ ComputeFeatureNeighborsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeFeatureNeighborsInputValues* inputValues); ~ComputeFeatureNeighborsScanline() noexcept; @@ -39,13 +46,27 @@ class SIMPLNXCORE_EXPORT ComputeFeatureNeighborsScanline ComputeFeatureNeighborsScanline& operator=(const ComputeFeatureNeighborsScanline&) = delete; ComputeFeatureNeighborsScanline& operator=(ComputeFeatureNeighborsScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized feature neighbor computation. + * + * Uses a 3-slice rolling window (prev/cur/next Z-slices) with bulk I/O: + * - copyIntoBuffer() reads one Z-slice of FeatureIds at a time + * - All 6 face-neighbor lookups are resolved from in-memory slice buffers + * - copyFromBuffer() writes the BoundaryCells output one Z-slice at a time + * + * The rolling window ensures only 3 slices of FeatureIds plus 1 slice of + * BoundaryCells are in memory at any time, regardless of volume size. + * Surface area accumulation uses per-face area values matching the Direct variant. + * + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const ComputeFeatureNeighborsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp index d38426c59d..44fb9b3c59 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.cpp @@ -17,6 +17,9 @@ using namespace nx::core; namespace { +/// Number of FeatureId tuples to read per bulk I/O call. 64K tuples balances +/// between minimizing copyIntoBuffer() round-trips and keeping per-chunk buffers +/// small enough to stay in CPU cache. constexpr usize k_ChunkTuples = 65536; constexpr int32 k_BadFeatureCount = -78231; constexpr uint64 k_MaxVoxelCount = std::numeric_limits::max(); @@ -41,7 +44,8 @@ Result<> ProcessImageGeom(ImageGeom& imageGeom, Float32AbstractDataStore& volume std::vector featureVoxelCounts(numFeatures, 0); msgHelper.sendMessage("Finding Voxel Counts..."); - // Count voxels per feature using chunked bulk I/O + // Count voxels per feature using chunked bulk I/O. Reading FeatureIds in 64K + // chunks via copyIntoBuffer() avoids per-element OOC chunk load/evict cycles. auto featureIdBuf = std::make_unique(k_ChunkTuples); for(usize offset = 0; offset < numVoxels; offset += k_ChunkTuples) { @@ -212,7 +216,9 @@ Result<> ProcessRectGridGeom(RectGridGeom& rectGridGeom, Float32AbstractDataStor std::vector featureCompensators(numFeatures, 0.0); msgHelper.sendMessage("Cell Level: Finding Voxel Counts and Summing Volumes..."); - // Count voxels and sum volumes using chunked bulk I/O + // Count voxels and sum volumes using chunked bulk I/O. For RectGrid, both + // FeatureIds and element sizes are read in lockstep chunks so that the + // per-element Kahan volume accumulation runs on local buffer data. auto featureIdBuf = std::make_unique(k_ChunkTuples); auto elemSizeBuf = std::make_unique(k_ChunkTuples); for(usize offset = 0; offset < numVoxels; offset += k_ChunkTuples) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.hpp index 40b1f053f3..1ef9bbd000 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeFeatureSizes.hpp @@ -11,30 +11,40 @@ #include "simplnx/Parameters/DataObjectNameParameter.hpp" #include "simplnx/Parameters/GeometrySelectionParameter.hpp" -/** -* This is example code to put in the Execute Method of the filter. - -*/ - namespace nx::core { +/** + * @struct ComputeFeatureSizesInputValues + * @brief Holds all user-configured parameters for the ComputeFeatureSizes algorithm. + */ struct SIMPLNXCORE_EXPORT ComputeFeatureSizesInputValues { - DataObjectNameParameter::ValueType EquivalentDiametersName; - AttributeMatrixSelectionParameter::ValueType FeatureAttributeMatrixPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - DataObjectNameParameter::ValueType NumElementsName; - BoolParameter::ValueType SaveElementSizes; - DataObjectNameParameter::ValueType VolumesName; + DataObjectNameParameter::ValueType EquivalentDiametersName; ///< Output: equivalent spherical/circular diameter array name. + AttributeMatrixSelectionParameter::ValueType FeatureAttributeMatrixPath; ///< Feature-level Attribute Matrix. + ArraySelectionParameter::ValueType FeatureIdsPath; ///< Per-cell Feature ID array. + GeometrySelectionParameter::ValueType InputImageGeometryPath; ///< Input ImageGeom or RectGridGeom. + DataObjectNameParameter::ValueType NumElementsName; ///< Output: per-feature voxel count array name. + BoolParameter::ValueType SaveElementSizes; ///< If true, persist per-element sizes in the Geometry. + DataObjectNameParameter::ValueType VolumesName; ///< Output: per-feature volume/area array name. }; /** * @class ComputeFeatureSizes - * @brief This algorithm implements support code for the ComputeFeatureSizesFilter + * @brief Computes the volume (or area in 2D), equivalent diameter, and voxel count + * for each feature in an Image Geometry or Rectilinear Grid Geometry. + * + * @section ooc_optimization Out-of-Core Optimization + * The original implementation iterated over all voxels using per-element getValue() + * calls on the FeatureIds and element-sizes DataStores. For OOC data this caused + * chunk thrashing on every voxel access. + * + * The optimized implementation reads FeatureIds (and element sizes for RectGrid) + * in fixed-size chunks (64K tuples) via copyIntoBuffer(), processing each chunk + * from a local buffer. Accumulation uses plain std::vectors, and Kahan summation + * for RectGrid volumes is performed on the local buffer data rather than through + * virtual DataStore dispatch. */ - class SIMPLNXCORE_EXPORT ComputeFeatureSizes { public: @@ -46,13 +56,17 @@ class SIMPLNXCORE_EXPORT ComputeFeatureSizes ComputeFeatureSizes& operator=(const ComputeFeatureSizes&) = delete; ComputeFeatureSizes& operator=(ComputeFeatureSizes&&) noexcept = delete; + /** + * @brief Executes the feature size computation using chunked bulk I/O. + * @return Result<> indicating success or error. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeFeatureSizesInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const ComputeFeatureSizesInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp index 7bcc1844f8..c074f57255 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.cpp @@ -8,6 +8,18 @@ using namespace nx::core; +// ============================================================================= +// ComputeKMedoids — Dispatcher +// +// This file contains only the dispatch logic. The actual algorithm implementations +// live in ComputeKMedoidsDirect.cpp (in-core) and ComputeKMedoidsScanline.cpp +// (out-of-core). +// +// The dispatch checks both the ClusteringArray and FeatureIds array storage types: +// if either uses chunked on-disk storage (OOC), the Scanline variant is selected +// to avoid chunk thrashing during the iterative distance computations. +// ============================================================================= + // ----------------------------------------------------------------------------- ComputeKMedoids::ComputeKMedoids(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues) : m_DataStructure(dataStructure) @@ -33,8 +45,19 @@ const std::atomic_bool& ComputeKMedoids::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to the appropriate algorithm variant based on storage type. + * + * Uses DispatchAlgorithm() to check whether the ClusteringArray + * or FeatureIds array is backed by out-of-core (chunked) storage. If so, the + * Scanline variant is used; otherwise, the Direct variant is selected. + * + * Both variants receive identical constructor arguments and produce identical output. + */ Result<> ComputeKMedoids::operator()() { + // Check both arrays — the clustering array is read repeatedly during distance + // computation, and featureIds is read/written during cluster assignment. auto* clusteringArray = m_DataStructure.getDataAs(m_InputValues->ClusteringArrayPath); auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); return DispatchAlgorithm({clusteringArray, featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp index 745b660209..235edaca2c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoids.hpp @@ -10,23 +10,64 @@ namespace nx::core { +/** + * @struct KMedoidsInputValues + * @brief Input parameter bundle for the ComputeKMedoids algorithm. + * + * Aggregates all DataPaths and configuration values needed by both the in-core + * (Direct) and out-of-core (Scanline) variants of K-Medoids clustering. + */ struct SIMPLNXCORE_EXPORT KMedoidsInputValues { - uint64 InitClusters; - ClusterUtilities::DistanceMetric DistanceMetric; - DataPath ClusteringArrayPath; - DataPath MaskArrayPath; - DataPath FeatureIdsArrayPath; - DataPath MedoidsArrayPath; - uint64 Seed; + uint64 InitClusters; ///< Number of clusters (k) to partition the data into + ClusterUtilities::DistanceMetric DistanceMetric; ///< Distance metric used for cluster assignment and medoid optimization + DataPath ClusteringArrayPath; ///< Input array containing the data to be clustered (any numeric type) + DataPath MaskArrayPath; ///< Input Bool/UInt8 mask array; false elements are assigned to cluster 0 + DataPath FeatureIdsArrayPath; ///< Output Int32 array storing per-element cluster assignments + DataPath MedoidsArrayPath; ///< Output array storing the medoid (representative point) for each cluster + uint64 Seed; ///< Random seed for reproducible initial medoid selection }; /** - * @class + * @class ComputeKMedoids + * @brief Dispatcher algorithm for K-Medoids clustering. + * + * K-Medoids is a partitioning clustering algorithm that assigns each data point to the + * nearest medoid (an actual data point that minimizes intra-cluster distance), then + * iteratively updates medoids until convergence. + * + * This class acts as a thin dispatcher that selects between two concrete implementations: + * + * - **ComputeKMedoidsDirect** (in-core): Uses per-element operator[] access for distance + * computation and cluster assignment. Optimal when all arrays reside in memory. + * + * - **ComputeKMedoidsScanline** (out-of-core / OOC): Uses chunked copyIntoBuffer() / + * copyFromBuffer() bulk I/O to read input data and write cluster assignments in + * fixed-size chunks (64K tuples), avoiding random per-element OOC access. + * + * The dispatch decision is made by DispatchAlgorithm() in + * AlgorithmDispatch.hpp, which checks whether any input IDataArray uses OOC storage. + * + * **Why two variants exist**: K-Medoids requires computing pairwise distances between + * data points and medoids. When data is stored out-of-core, each operator[] access may + * trigger a chunk load/evict cycle. The Scanline variant caches medoids locally and + * processes the input array in sequential 64K-tuple chunks, converting random access + * into sequential bulk reads. + * + * @see ComputeKMedoidsDirect + * @see ComputeKMedoidsScanline + * @see AlgorithmDispatch.hpp */ class SIMPLNXCORE_EXPORT ComputeKMedoids { public: + /** + * @brief Constructs the dispatcher with all resources needed by either algorithm variant. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ ComputeKMedoids(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, KMedoidsInputValues* inputValues); ~ComputeKMedoids() noexcept; @@ -35,15 +76,29 @@ class SIMPLNXCORE_EXPORT ComputeKMedoids ComputeKMedoids& operator=(const ComputeKMedoids&) = delete; ComputeKMedoids& operator=(ComputeKMedoids&&) noexcept = delete; + /** + * @brief Dispatches to the Direct or Scanline algorithm based on storage type. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); + + /** + * @brief Sends a progress message through the filter's message handler. + * @param message The progress message text + */ void updateProgress(const std::string& message); + + /** + * @brief Returns a reference to the cancellation flag for checking in inner loops. + * @return Reference to the atomic bool cancellation flag + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const KMedoidsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const KMedoidsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp index 0c5bae22f8..a18887021a 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.cpp @@ -13,8 +13,39 @@ using namespace nx::core; +// ============================================================================= +// ComputeKMedoidsDirect — In-Core Algorithm +// +// This file implements the in-core (Direct) variant of ComputeKMedoids. +// It is selected by DispatchAlgorithm when all input arrays reside in memory. +// +// ALGORITHM OVERVIEW (Voronoi Iteration / PAM): +// 1. Randomly select k initial medoids from masked data points +// 2. Assign each point to the nearest medoid (findClusters) +// 3. For each cluster, find the member that minimizes total intra-cluster +// distance — this becomes the new medoid (optimizeClusters) +// 4. Repeat steps 2-3 until medoids stop changing (convergence) +// +// DATA ACCESS PATTERN: +// Uses operator[] for per-element random access to the input array, medoids +// array, and featureIds array. This is optimal for in-memory DataStore where +// operator[] is essentially a pointer dereference. For out-of-core data, this +// pattern would cause chunk thrashing — see ComputeKMedoidsScanline instead. +// +// COMPLEXITY: +// findClusters: O(n * k * d) per iteration +// optimizeClusters: O(k * n_i^2 * d) per iteration, where n_i is cluster size +// Total: O(iter * (n*k*d + k*n_i^2*d)) +// ============================================================================= + namespace { +/** + * @brief Type-specialized template that performs the actual K-Medoids computation + * for the in-core (Direct) path. + * + * @tparam T The element type of the clustering array (e.g., float32, int32) + */ template class KMedoidsTemplate { @@ -37,6 +68,10 @@ class KMedoidsTemplate void operator=(const KMedoidsTemplate&) = delete; // Move assignment Not Implemented // ----------------------------------------------------------------------------- + /** + * @brief Main K-Medoids loop: initialize medoids, then iterate findClusters + + * optimizeClusters until convergence (medoid indices stop changing). + */ void operator()() { usize numTuples = m_InputArray.getNumberOfTuples(); @@ -109,6 +144,16 @@ class KMedoidsTemplate std::mt19937_64::result_type m_Seed; // ----------------------------------------------------------------------------- + /** + * @brief Assigns each data point to the nearest medoid using direct operator[] access. + * + * For each masked data point, computes the distance to all k medoids and assigns + * the point to the cluster of the nearest medoid. Uses direct per-element access + * via operator[] — optimal for in-memory data but would cause chunk thrashing for OOC. + * + * @param tuples Total number of tuples in the input array + * @param dims Number of components per tuple + */ void findClusters(usize tuples, int32 dims) { for(usize i = 0; i < tuples; i++) @@ -134,6 +179,21 @@ class KMedoidsTemplate } // ----------------------------------------------------------------------------- + /** + * @brief Finds the optimal medoid for each cluster by minimizing total intra-cluster distance. + * + * For each cluster i, iterates over all members j of that cluster. For each candidate + * medoid j, computes the total distance from j to all other members k. The member with + * the lowest total cost becomes the new medoid. + * + * Complexity: O(k * n_i^2 * dims) where n_i is the size of cluster i. + * Uses direct per-element access via operator[]. + * + * @param tuples Total number of tuples in the input array + * @param dims Number of components per tuple + * @param clusterIdxs In/out: current medoid indices, updated with new optimal medoids + * @return Per-cluster minimum cost vector + */ std::vector optimizeClusters(usize tuples, int32 dims, std::vector& clusterIdxs) { std::vector minCosts(m_NumClusters, std::numeric_limits::max()); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp index d4cbba8441..e2395410f3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsDirect.hpp @@ -11,13 +11,33 @@ struct KMedoidsInputValues; /** * @class ComputeKMedoidsDirect - * @brief In-core algorithm for ComputeKMedoids. Uses direct per-element operator[] - * access for distance computation, cluster assignment, and medoid optimization. + * @brief In-core algorithm for K-Medoids clustering using direct per-element array access. + * + * Uses operator[] for distance computation, cluster assignment, and medoid optimization. + * This is optimal when all arrays reside in memory, where operator[] is essentially a + * pointer dereference. + * + * The algorithm uses Voronoi iteration: + * 1. Randomly select k initial medoids from masked data points + * 2. Assign each point to the nearest medoid (findClusters) + * 3. For each cluster, find the member that minimizes total intra-cluster distance (optimizeClusters) + * 4. Repeat steps 2-3 until medoids stop changing + * * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * + * @see ComputeKMedoidsScanline for the out-of-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT ComputeKMedoidsDirect { public: + /** + * @brief Constructs the in-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ ComputeKMedoidsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const KMedoidsInputValues* inputValues); ~ComputeKMedoidsDirect() noexcept; @@ -26,16 +46,29 @@ class SIMPLNXCORE_EXPORT ComputeKMedoidsDirect ComputeKMedoidsDirect& operator=(const ComputeKMedoidsDirect&) = delete; ComputeKMedoidsDirect& operator=(ComputeKMedoidsDirect&&) noexcept = delete; + /** + * @brief Executes the in-core K-Medoids clustering. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); + /** + * @brief Sends a progress message through the filter's message handler. + * @param message The progress message text + */ void updateProgress(const std::string& message); + + /** + * @brief Returns a reference to the cancellation flag for checking in inner loops. + * @return Reference to the atomic bool cancellation flag + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const KMedoidsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const KMedoidsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp index f9b0b1004b..c828be0141 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.cpp @@ -13,8 +13,50 @@ using namespace nx::core; +// ============================================================================= +// ComputeKMedoidsScanline — Out-of-Core (OOC) Algorithm +// +// This file implements the out-of-core (Scanline) variant of ComputeKMedoids. +// It is selected by DispatchAlgorithm when any input array uses chunked on-disk +// storage (e.g., ZarrStore / HDF5 chunked store). +// +// PROBLEM: +// K-Medoids requires repeated passes over the input data: +// - findClusters: reads all N tuples to compute distances to k medoids +// - optimizeClusters: reads cluster members pairwise to find optimal medoids +// With OOC storage, each operator[] access may load an entire chunk from disk, +// use one value, then evict it. Over millions of tuples and multiple iterations, +// this chunk thrashing makes the algorithm catastrophically slow. +// +// SOLUTION — CHUNKED BULK I/O: +// Instead of per-element random access, read data in sequential 64K-tuple chunks +// using copyIntoBuffer(). This aligns with OOC chunk boundaries and amortizes +// the cost of disk I/O across thousands of elements per read. +// +// KEY OOC OPTIMIZATIONS: +// 1. Medoid cache: The medoids array is tiny (k * numComponents), so it is cached +// entirely in a local std::vector before each findClusters pass. +// 2. Chunked assignment: Input data and featureIds are read/written in aligned +// 64K-tuple chunks. All distance computations for each chunk are done in +// memory before writing the updated featureIds back. +// 3. Per-cluster member scanning: optimizeClusters scans featureIds in chunks +// to build a member index list for one cluster at a time, keeping peak +// memory at O(max_cluster_size) instead of O(n). +// 4. Per-tuple reads for pairwise distances: Each candidate medoid and comparison +// member is read via single-tuple copyIntoBuffer() calls. While this is still +// O(n_i^2) reads per cluster, each read is a known-location bulk I/O call +// rather than a virtual operator[] dispatch, and the grid of reads is bounded +// by cluster size rather than total array size. +// ============================================================================= + namespace { +/** + * @brief Type-specialized template that performs the actual K-Medoids computation + * for the out-of-core (Scanline) path using chunked bulk I/O. + * + * @tparam T The element type of the clustering array (e.g., float32, int32) + */ template class KMedoidsTemplate { @@ -37,6 +79,13 @@ class KMedoidsTemplate void operator=(const KMedoidsTemplate&) = delete; // Move assignment Not Implemented // ----------------------------------------------------------------------------- + /** + * @brief Main K-Medoids loop: initialize medoids via bulk I/O, then iterate + * findClusters + optimizeClusters until convergence. + * + * Medoid initialization uses copyIntoBuffer()/copyFromBuffer() per-tuple instead + * of operator[] to avoid OOC random access during setup. + */ void operator()() { usize numTuples = m_InputArray.getNumberOfTuples(); @@ -58,7 +107,10 @@ class KMedoidsTemplate } } - // OOC: use bulk I/O to initialize medoids + // OOC: use bulk I/O to initialize medoids. Each medoid is a single tuple + // read from a random position in the input array. Using copyIntoBuffer() + // instead of operator[] ensures we go through the bulk I/O path, which + // reads a known-size chunk from disk rather than triggering virtual dispatch. auto tupleBuf = std::make_unique(numCompDims); for(usize i = 0; i < m_NumClusters; i++) { @@ -109,10 +161,22 @@ class KMedoidsTemplate std::mt19937_64::result_type m_Seed; // ----------------------------------------------------------------------------- - // OOC: cache medoids locally, process input and featureIds in chunks + /** + * @brief OOC-optimized cluster assignment: cache medoids locally, then process + * the input array and featureIds in aligned 64K-tuple chunks. + * + * Why cache medoids: The medoids array is tiny (k * dims elements), so caching + * it in a local vector eliminates k * n per-element OOC reads per iteration. + * The input data and featureIds are then read/written in sequential chunks, + * converting O(n) random accesses into O(n / chunkSize) bulk I/O calls. + * + * @param tuples Total number of tuples in the input array + * @param dims Number of components per tuple + */ void findClusters(usize tuples, int32 dims) { - // Cache medoids (small: numClusters * dims) + // Cache the entire medoids array in memory. This is small (typically k < 100 + // and dims < 10, so < 1 KB) and avoids repeated OOC reads during the inner loop. const usize medoidsSize = (m_NumClusters + 1) * dims; std::vector medoidsCache(medoidsSize); m_Medoids.copyIntoBuffer(0, nonstd::span(medoidsCache.data(), medoidsSize)); @@ -155,8 +219,24 @@ class KMedoidsTemplate } // ----------------------------------------------------------------------------- - // OOC: process one cluster at a time to avoid O(n) total member list allocation. - // Peak memory is O(max_cluster_size), not O(n). + /** + * @brief OOC-optimized medoid optimization: process one cluster at a time. + * + * Instead of building a global member list for all clusters (O(n) memory), this + * processes clusters sequentially. For each cluster: + * 1. Scan featureIds in 64K-tuple chunks to collect member indices + * 2. For each candidate medoid j in the cluster, compute total distance to + * all other members k using per-tuple copyIntoBuffer() reads + * 3. The member with the lowest total cost becomes the new medoid + * 4. Release the member list before processing the next cluster + * + * Peak memory is O(max_cluster_size) per cluster, not O(n) for all clusters. + * + * @param tuples Total number of tuples in the input array + * @param dims Number of components per tuple + * @param clusterIdxs In/out: current medoid indices, updated with new optimal medoids + * @return Per-cluster minimum cost vector + */ std::vector optimizeClusters(usize tuples, int32 dims, std::vector& clusterIdxs) { std::vector minCosts(m_NumClusters, std::numeric_limits::max()); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp index e2af49976e..65a3952ea4 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeKMedoidsScanline.hpp @@ -11,13 +11,43 @@ struct KMedoidsInputValues; /** * @class ComputeKMedoidsScanline - * @brief Out-of-core algorithm for ComputeKMedoids. Uses chunked copyIntoBuffer/copyFromBuffer - * bulk I/O for distance computation, cluster assignment, and medoid optimization. + * @brief Out-of-core algorithm for K-Medoids clustering using chunked bulk I/O. + * + * Uses copyIntoBuffer()/copyFromBuffer() to read input data and write cluster + * assignments in fixed-size chunks (64K tuples), avoiding random per-element + * OOC access that would cause chunk thrashing. + * + * Key OOC optimizations over the Direct variant: + * + * - **Medoid initialization**: Uses copyIntoBuffer()/copyFromBuffer() per-tuple + * instead of operator[] to read initial medoid values. + * + * - **Cluster assignment (findClusters)**: Caches all medoids in a local vector + * (small: k * numComponents), then processes the input array and featureIds + * in aligned 64K-tuple chunks via bulk I/O. Each chunk is read once, all + * distance computations for that chunk are done in memory, then featureIds + * are written back in one bulk operation. + * + * - **Medoid optimization (optimizeClusters)**: Processes one cluster at a time. + * Scans featureIds in chunks to build a member index list, then computes + * pairwise distances using per-tuple copyIntoBuffer() reads. Peak memory is + * O(max_cluster_size), not O(n). + * * Selected by DispatchAlgorithm when any input array is backed by out-of-core storage. + * + * @see ComputeKMedoidsDirect for the in-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT ComputeKMedoidsScanline { public: + /** + * @brief Constructs the out-of-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ ComputeKMedoidsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const KMedoidsInputValues* inputValues); ~ComputeKMedoidsScanline() noexcept; @@ -26,16 +56,29 @@ class SIMPLNXCORE_EXPORT ComputeKMedoidsScanline ComputeKMedoidsScanline& operator=(const ComputeKMedoidsScanline&) = delete; ComputeKMedoidsScanline& operator=(ComputeKMedoidsScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized K-Medoids clustering. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); + /** + * @brief Sends a progress message through the filter's message handler. + * @param message The progress message text + */ void updateProgress(const std::string& message); + + /** + * @brief Returns a reference to the cancellation flag for checking in inner loops. + * @return Reference to the atomic bool cancellation flag + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const KMedoidsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const KMedoidsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp index 155fb3188a..c9e5783584 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.cpp @@ -8,6 +8,24 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeSurfaceAreaToVolume -- Dispatcher +// +// This file implements the thin dispatch layer for the ComputeSurfaceAreaToVolume +// algorithm. No algorithm logic lives here; the sole responsibility is to +// inspect the storage type of the FeatureIds array and forward execution to +// either ComputeSurfaceAreaToVolumeDirect (in-core) or +// ComputeSurfaceAreaToVolumeScanline (out-of-core), via the DispatchAlgorithm +// template. +// +// The FeatureIds array is the critical input for dispatch because it is a +// cell-level array (one entry per voxel) that is accessed with 6-neighbor +// lookups. The feature-level arrays (NumCells, SurfaceAreaVolumeRatio, +// Sphericity) are small and do not drive the dispatch decision. The Scanline +// variant still caches these locally for efficiency, but the dispatch is based +// solely on FeatureIds. +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeSurfaceAreaToVolume::ComputeSurfaceAreaToVolume(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues) @@ -28,6 +46,16 @@ const std::atomic_bool& ComputeSurfaceAreaToVolume::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Inspects the FeatureIds array's storage type and dispatches to the + * appropriate algorithm variant. + * + * The dispatch decision is made by DispatchAlgorithm, which checks: + * 1. ForceInCoreAlgorithm() -- test override, always selects Direct + * 2. AnyOutOfCore({featureIdsArray}) -- runtime detection of chunked storage + * 3. ForceOocAlgorithm() -- test override, forces Scanline + * 4. Default -- selects Direct (in-core) + */ Result<> ComputeSurfaceAreaToVolume::operator()() { auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp index 637caf9980..b540978df5 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolume.hpp @@ -9,22 +9,56 @@ namespace nx::core { +/** + * @struct ComputeSurfaceAreaToVolumeInputValues + * @brief Holds all user-configurable parameters for the ComputeSurfaceAreaToVolume algorithm. + * + * These values are extracted from the filter's parameter map and passed through + * the dispatcher to whichever algorithm variant (Direct or Scanline) is selected. + */ struct SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeInputValues { - DataPath FeatureIdsArrayPath; - DataPath NumCellsArrayPath; - DataPath SurfaceAreaVolumeRatioArrayName; - bool CalculateSphericity; - DataPath SphericityArrayName; - DataPath InputImageGeometry; + DataPath FeatureIdsArrayPath; ///< Path to the cell-level Int32 FeatureIds array. + DataPath NumCellsArrayPath; ///< Path to the feature-level Int32 NumCells array (voxel count per feature). + DataPath SurfaceAreaVolumeRatioArrayName; ///< Path where the output Float32 SA/V ratio array will be stored. + bool CalculateSphericity; ///< When true, also computes the sphericity for each feature. + DataPath SphericityArrayName; ///< Path where the output Float32 sphericity array will be stored (only used when CalculateSphericity is true). + DataPath InputImageGeometry; ///< Path to the ImageGeom that defines grid dimensions and voxel spacing. }; /** - * @class + * @class ComputeSurfaceAreaToVolume + * @brief Dispatcher that selects between the in-core (Direct) and out-of-core (Scanline) + * surface-area-to-volume ratio algorithms at runtime. + * + * This class does not contain any algorithm logic itself. Its operator()() inspects + * the storage backing of the FeatureIds array and calls + * `DispatchAlgorithm(...)`. + * + * **Algorithm overview**: For each voxel in the image geometry, the algorithm examines + * its 6 face neighbors. Whenever a neighbor belongs to a different feature, the shared + * face area is added to the current feature's surface area accumulator. After processing + * all voxels, the surface area is divided by the feature volume (numCells * voxelVolume) + * to produce the SA/V ratio. Optionally, sphericity is also computed from the same + * surface area and volume values. + * + * **Dispatch rules** (see AlgorithmDispatch.hpp): + * - If all input arrays are in-memory, the Direct variant is selected. + * - If any input array uses OOC storage, the Scanline variant is selected. + * - Test-override flags can force either path. + * + * @see ComputeSurfaceAreaToVolumeDirect, ComputeSurfaceAreaToVolumeScanline, DispatchAlgorithm */ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolume { public: + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all arrays and geometries. + * @param mesgHandler Handler for sending progress/info messages back to the UI. + * @param shouldCancel Atomic flag checked periodically to support user cancellation. + * @param inputValues User-configured parameters for the algorithm. + */ ComputeSurfaceAreaToVolume(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceAreaToVolumeInputValues* inputValues); ~ComputeSurfaceAreaToVolume() noexcept; @@ -33,15 +67,24 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolume ComputeSurfaceAreaToVolume& operator=(const ComputeSurfaceAreaToVolume&) = delete; ComputeSurfaceAreaToVolume& operator=(ComputeSurfaceAreaToVolume&&) noexcept = delete; + /** + * @brief Dispatches to the appropriate algorithm variant (Direct or Scanline) + * based on whether the FeatureIds array uses out-of-core storage. + * @return Result<> indicating success or any errors encountered. + */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag. + * @return Const reference to the atomic cancellation boolean. + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; ///< User-configured algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Atomic flag for cooperative cancellation. + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress and informational messages. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp index 923e17b67b..7f009f00a3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.cpp @@ -12,6 +12,31 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeSurfaceAreaToVolumeDirect -- In-Core Algorithm +// +// Computes the surface-area-to-volume ratio (and optional sphericity) for each +// feature in an image geometry. The algorithm has two phases: +// +// Phase 1 -- Surface area accumulation: +// Iterate every voxel in Z-Y-X order. For each voxel with FeatureId > 0, +// check its 6 face neighbors. When a neighbor belongs to a different feature, +// the area of the shared face is added to the current feature's accumulator. +// Face areas depend on the voxel spacing: +// - Z-normal faces (shared by +/-Z neighbors): spacing[0] * spacing[1] +// - Y-normal faces (shared by +/-Y neighbors): spacing[1] * spacing[2] +// - X-normal faces (shared by +/-X neighbors): spacing[2] * spacing[0] +// +// Phase 2 -- Ratio and sphericity computation: +// For each feature, divide accumulated surface area by feature volume +// (numCells * voxelVolume). Optionally compute sphericity using: +// sphericity = (pi^(1/3) * (6*V)^(2/3)) / SA +// +// Data access pattern: Uses operator[] on the FeatureIds DataStore with +// pre-computed flat-index offsets for the 6 face neighbors. This is efficient +// for in-memory data but would cause chunk thrashing on OOC storage. +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeSurfaceAreaToVolumeDirect::ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceAreaToVolumeInputValues* inputValues) @@ -27,25 +52,43 @@ ComputeSurfaceAreaToVolumeDirect::~ComputeSurfaceAreaToVolumeDirect() noexcept = // ----------------------------------------------------------------------------- /** - * @brief Computes surface-area-to-volume ratio using direct Z-Y-X iteration. - * In-core path: accumulates per-feature surface area from face-neighbor - * comparisons, then divides by voxel volume. Optionally computes sphericity. + * @brief Computes surface-area-to-volume ratio (and optional sphericity) using + * direct in-memory array indexing. + * + * The algorithm proceeds in two phases: + * + * **Phase 1 -- Surface area accumulation** (voxel-level): + * For each voxel in Z-Y-X order, check 6 face neighbors via flat-index offsets. + * Boundary neighbors (outside the volume) are skipped. When a valid neighbor + * belongs to a different feature, the shared face area is added to the current + * feature's surface-area accumulator. The face area depends on which axis the + * face is normal to (Z-normal = spacing.x * spacing.y, etc.). + * + * **Phase 2 -- Ratio computation** (feature-level): + * For each feature (starting from feature 1, since feature 0 is background): + * - Compute volume = numCells * voxelVolume + * - SA/V ratio = surfaceArea / volume + * - Sphericity (optional) = (pi^(1/3) * (6*V)^(2/3)) / SA + * + * @return Result<> indicating success, validation errors, or cancellation. */ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() { - // Input Cell Data + // -- Setup: Retrieve input arrays and geometry -- + + // Cell-level FeatureIds: one int32 per voxel identifying which feature owns it auto featureIdsArrayPtr = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); const auto& featureIdsStoreRef = featureIdsArrayPtr->getDataStoreRef(); - // Input Feature Data + // Feature-level NumCells: pre-computed count of how many voxels each feature has const auto& numCells = m_DataStructure.getDataAs(m_InputValues->NumCellsArrayPath)->getDataStoreRef(); - // Output Feature Data + // Output: SA/V ratio per feature auto& surfaceAreaVolumeRatio = m_DataStructure.getDataAs(m_InputValues->SurfaceAreaVolumeRatioArrayName)->getDataStoreRef(); - // Required Geometry const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometry); + // Validate that the max FeatureId does not exceed the feature AttributeMatrix size auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, m_InputValues->NumCellsArrayPath.getParent(), *featureIdsArrayPtr, false, m_MessageHandler); if(validateNumFeatResult.invalid()) { @@ -59,10 +102,22 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() auto yPoints = static_cast(dims[1]); auto zPoints = static_cast(dims[2]); + // Volume of a single voxel, used to convert numCells to physical volume float32 voxelVol = spacing[0] * spacing[1] * spacing[2]; + // Local accumulator for per-feature surface area. Using a std::vector here + // (rather than the output DataStore) because multiple voxels contribute to + // the same feature and we need read-modify-write access during accumulation. std::vector featureSurfaceArea(static_cast(numFeatures), 0.0f); + // Pre-compute flat-index offsets for the 6 face neighbors. + // For a voxel at flat index i: + // -Z neighbor: i - (xPoints * yPoints) (one full Z-slice back) + // -Y neighbor: i - xPoints (one row back) + // -X neighbor: i - 1 (one element back) + // +X neighbor: i + 1 (one element forward) + // +Y neighbor: i + xPoints (one row forward) + // +Z neighbor: i + (xPoints * yPoints) (one full Z-slice forward) int64 neighborOffset[6] = {0, 0, 0, 0, 0, 0}; neighborOffset[0] = -xPoints * yPoints; // -Z neighborOffset[1] = -xPoints; // -Y @@ -71,6 +126,9 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() neighborOffset[4] = xPoints; // +Y neighborOffset[5] = xPoints * yPoints; // +Z + // -- Phase 1: Surface area accumulation -- + // Iterate every voxel, check 6 neighbors, accumulate shared face areas. + for(int64 zIdx = 0; zIdx < zPoints; zIdx++) { if(m_ShouldCancel) @@ -85,13 +143,17 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() { float32 onSurface = 0.0f; int32 currentFeatureId = featureIdsStoreRef[zStride + yStride + xIdx]; + // Skip background voxels (FeatureId <= 0) if(currentFeatureId < 1) { continue; } + // Check each of the 6 face neighbors. Skip neighbors that would be + // outside the volume (boundary voxels have fewer valid neighbors). for(int32 neighborOffsetIndex = 0; neighborOffsetIndex < 6; neighborOffsetIndex++) { + // Boundary guards: skip neighbor if it would be out of bounds if(neighborOffsetIndex == 0 && zIdx == 0) { continue; @@ -119,28 +181,35 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() int64 neighborIndex = zStride + yStride + xIdx + neighborOffset[neighborOffsetIndex]; + // If the neighbor belongs to a different feature, the shared face + // contributes to this feature's surface area. The face area depends + // on which axis the face is normal to. if(featureIdsStoreRef[neighborIndex] != currentFeatureId) { - if(neighborOffsetIndex == 0 || neighborOffsetIndex == 5) // XY face shared + if(neighborOffsetIndex == 0 || neighborOffsetIndex == 5) // Z-normal face (XY plane) { onSurface = onSurface + spacing[0] * spacing[1]; } - if(neighborOffsetIndex == 1 || neighborOffsetIndex == 4) // YZ face shared + if(neighborOffsetIndex == 1 || neighborOffsetIndex == 4) // Y-normal face (XZ plane) { onSurface = onSurface + spacing[1] * spacing[2]; } - if(neighborOffsetIndex == 2 || neighborOffsetIndex == 3) // XZ face shared + if(neighborOffsetIndex == 2 || neighborOffsetIndex == 3) // X-normal face (YZ plane) { onSurface = onSurface + spacing[2] * spacing[0]; } } } + // Add this voxel's boundary face contributions to the feature total int32 featureId = featureIdsStoreRef[zStride + yStride + xIdx]; featureSurfaceArea[featureId] = featureSurfaceArea[featureId] + onSurface; } } } + // -- Phase 2: Ratio and sphericity computation -- + + // Compute SA/V ratio for each feature (skip feature 0 = background) const float32 thirdRootPi = std::pow(nx::core::Constants::k_PiF, 0.333333f); for(usize i = 1; i < numFeatures; i++) { @@ -148,6 +217,9 @@ Result<> ComputeSurfaceAreaToVolumeDirect::operator()() surfaceAreaVolumeRatio[i] = featureSurfaceArea[i] / featureVolume; } + // Optionally compute sphericity: how close each feature's shape is to a sphere. + // Sphericity = (pi^(1/3) * (6V)^(2/3)) / SA + // A perfect sphere has sphericity = 1.0; more irregular shapes have lower values. if(m_InputValues->CalculateSphericity) { m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Computing Sphericity")); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp index 54e3160442..3557e17d08 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeDirect.hpp @@ -11,13 +11,46 @@ struct ComputeSurfaceAreaToVolumeInputValues; /** * @class ComputeSurfaceAreaToVolumeDirect - * @brief In-core algorithm for ComputeSurfaceAreaToVolume. Preserves the original sequential - * Z-Y-X voxel iteration with face-neighbor surface area accumulation per feature. - * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * @brief In-core (direct memory access) algorithm for computing per-feature surface area, + * volume, surface-area-to-volume ratio, and optional sphericity. + * + * This is the traditional algorithm that uses operator[] to read FeatureIds directly + * through the DataStore abstraction. It iterates all voxels in Z-Y-X order and, for + * each voxel, checks its 6 face neighbors using pre-computed index offsets. When a + * neighbor belongs to a different feature, the area of the shared face (determined by + * the voxel spacing along each axis) is added to the current feature's surface-area + * accumulator. After the full-volume scan, the ratio and optional sphericity are + * computed per feature. + * + * **When this variant is selected**: DispatchAlgorithm selects this class when all + * input arrays are backed by contiguous in-memory DataStore. With in-memory data, + * the neighbor lookups via flat-index offsets are simple pointer arithmetic. + * + * **Why a separate OOC variant exists**: The 6-neighbor lookup pattern accesses + * elements at offsets of +/-1, +/-dimX, and +/-(dimX*dimY) from the current voxel. + * When FeatureIds is stored out-of-core in chunked format, these scattered accesses + * cause chunk thrashing. The Scanline variant reads entire Z-slices sequentially + * with copyIntoBuffer() to avoid this. + * + * **Output details**: The per-feature surface-area accumulation uses a local + * std::vector (not the output DataStore) because multiple voxels + * contribute to the same feature's surface area. After the scan, the SA/V ratio + * is computed and written to the output array. Sphericity (if requested) uses + * the formula: sphericity = (pi^(1/3) * (6V)^(2/3)) / SA. + * + * @see ComputeSurfaceAreaToVolumeScanline for the OOC-optimized variant. + * @see ComputeSurfaceAreaToVolume for the dispatcher. */ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeDirect { public: + /** + * @brief Constructs the in-core SA/V ratio calculator. + * @param dataStructure The DataStructure containing all arrays and the ImageGeom. + * @param mesgHandler Handler for progress/info messages. + * @param shouldCancel Atomic flag for cooperative cancellation. + * @param inputValues Algorithm parameters (geometry path, array paths, sphericity flag). + */ ComputeSurfaceAreaToVolumeDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceAreaToVolumeInputValues* inputValues); ~ComputeSurfaceAreaToVolumeDirect() noexcept; @@ -27,13 +60,22 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeDirect ComputeSurfaceAreaToVolumeDirect& operator=(const ComputeSurfaceAreaToVolumeDirect&) = delete; ComputeSurfaceAreaToVolumeDirect& operator=(ComputeSurfaceAreaToVolumeDirect&&) noexcept = delete; + /** + * @brief Executes the in-core surface-area-to-volume ratio algorithm. + * + * Iterates every voxel in Z-Y-X order, accumulates per-feature surface area + * from face-neighbor comparisons, then divides by feature volume. Optionally + * computes sphericity. + * + * @return Result<> indicating success or errors. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; ///< Algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cooperative cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message handler. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp index f53f25c0bb..928a6b4900 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.cpp @@ -13,6 +13,43 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeSurfaceAreaToVolumeScanline -- Out-of-Core Algorithm +// +// Computes the surface-area-to-volume ratio (and optional sphericity) for each +// feature in an image geometry. Produces identical results to the Direct variant +// but uses a 3-slice rolling window for sequential bulk I/O. +// +// KEY DESIGN PRINCIPLE: ALL array access uses bulk I/O (copyIntoBuffer / +// copyFromBuffer). There are zero operator[] calls on any DataStore. +// +// The algorithm has three phases: +// +// Phase 1 -- Surface area accumulation via Z-slice rolling window: +// Three std::vector buffers (prevSlice, curSlice, nextSlice) hold +// adjacent Z-slices of FeatureIds. For each voxel in curSlice, the 6 +// neighbors are checked using these buffers: +// - -Z / +Z: prevSlice[inSlice] / nextSlice[inSlice] +// - -Y / +Y: curSlice[inSlice - xPoints] / curSlice[inSlice + xPoints] +// - -X / +X: curSlice[inSlice - 1] / curSlice[inSlice + 1] +// Shared face areas are accumulated into a local std::vector. +// +// Phase 2 -- Ratio computation with local caches: +// The feature-level NumCells array is bulk-read into a local vector, the +// SA/V ratio is computed locally, and the result is bulk-written back. +// +// Phase 3 -- Optional sphericity computation: +// Same local-cache approach as Phase 2. +// +// MEMORY BUDGET: +// - 3 Z-slice buffers: 3 * dimX * dimY * 4 bytes +// - featureSurfaceArea: numFeatures * 4 bytes +// - localNumCells: numFeatures * 4 bytes +// - localSurfaceAreaVolumeRatio: numFeatures * 4 bytes +// - localSphericity (if needed): numFeatures * 4 bytes +// Total: ~12 * dimX * dimY + ~16 * numFeatures bytes +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeSurfaceAreaToVolumeScanline::ComputeSurfaceAreaToVolumeScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceAreaToVolumeInputValues* inputValues) @@ -28,26 +65,48 @@ ComputeSurfaceAreaToVolumeScanline::~ComputeSurfaceAreaToVolumeScanline() noexce // ----------------------------------------------------------------------------- /** - * @brief Computes surface-area-to-volume ratio using Z-slice bulk I/O with a rolling window. - * OOC path: reads featureIds one Z-slice at a time via copyIntoBuffer(), - * keeping 3 slices resident (prev/cur/next) for cross-slice neighbour access. - * Same logic as ComputeSurfaceAreaToVolumeDirect. + * @brief Computes surface-area-to-volume ratio (and optional sphericity) using a + * 3-slice rolling window with sequential bulk I/O for out-of-core storage + * compatibility. + * + * **Phase 1 -- Surface area accumulation**: + * - Initialize the rolling window by loading Z-slices 0 and 1. + * - For each Z-slice, iterate all voxels in Y-X order within curSlice. + * - For each voxel with FeatureId > 0, check 6 face neighbors: + * - -Z: prevSlice[inSlice], +Z: nextSlice[inSlice] + * - -Y: curSlice[inSlice - xPoints], +Y: curSlice[inSlice + xPoints] + * - -X: curSlice[inSlice - 1], +X: curSlice[inSlice + 1] + * - When a neighbor differs, add the appropriate face area. + * - After each Z-slice, rotate buffers and load the next slice. + * + * **Phase 2 -- Ratio computation**: + * - Bulk-read the feature-level NumCells array into a local vector. + * - Compute SA/V = featureSurfaceArea[i] / (voxelVol * numCells[i]). + * - Bulk-write the ratio array back to the OOC store. + * + * **Phase 3 -- Optional sphericity**: + * - Compute sphericity = (pi^(1/3) * (6V)^(2/3)) / SA in a local buffer. + * - Bulk-write the sphericity array back to the OOC store. + * + * @return Result<> indicating success, validation errors, or cancellation. */ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() { - // Input Cell Data + // -- Setup: Retrieve input arrays and geometry -- + + // Cell-level FeatureIds: accessed via rolling window, never via operator[] auto featureIdsArrayPtr = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); auto& featureIdsStore = featureIdsArrayPtr->getDataStoreRef(); - // Input Feature Data + // Feature-level NumCells: will be bulk-read into a local vector in Phase 2 const auto& numCells = m_DataStructure.getDataAs(m_InputValues->NumCellsArrayPath)->getDataStoreRef(); - // Output Feature Data + // Output SA/V ratio: will be bulk-written from a local vector in Phase 2 auto& surfaceAreaVolumeRatio = m_DataStructure.getDataAs(m_InputValues->SurfaceAreaVolumeRatioArrayName)->getDataStoreRef(); - // Required Geometry const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->InputImageGeometry); + // Validate that the max FeatureId does not exceed the feature AttributeMatrix size auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, m_InputValues->NumCellsArrayPath.getParent(), *featureIdsArrayPtr, false, m_MessageHandler); if(validateNumFeatResult.invalid()) { @@ -61,23 +120,34 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() auto yPoints = static_cast(dims[1]); auto zPoints = static_cast(dims[2]); + // Volume of a single voxel, used to convert numCells to physical volume float32 voxelVol = spacing[0] * spacing[1] * spacing[2]; + // Local accumulator for per-feature surface area std::vector featureSurfaceArea(static_cast(numFeatures), 0.0f); - // Face areas for each neighbor direction - const float32 xyFaceArea = spacing[0] * spacing[1]; // XY face shared (Z-normal) - const float32 yzFaceArea = spacing[1] * spacing[2]; // YZ face shared (X-normal) - const float32 zxFaceArea = spacing[2] * spacing[0]; // XZ face shared (Y-normal) + // Pre-compute face areas for each neighbor direction. These depend on the + // voxel spacing and which axis the shared face is normal to: + // - Z-normal face (shared by +/-Z neighbors): area = spacing.x * spacing.y + // - X-normal face (shared by +/-X neighbors): area = spacing.y * spacing.z + // - Y-normal face (shared by +/-Y neighbors): area = spacing.z * spacing.x + const float32 xyFaceArea = spacing[0] * spacing[1]; // Z-normal face area + const float32 yzFaceArea = spacing[1] * spacing[2]; // X-normal face area + const float32 zxFaceArea = spacing[2] * spacing[0]; // Y-normal face area + + // -- Phase 1: Surface area accumulation via Z-slice rolling window -- - // Z-slice rolling window for bulk I/O + // Each Z-slice has yPoints * xPoints voxels const usize sliceSize = static_cast(yPoints) * static_cast(xPoints); + + // Allocate the 3-slice rolling window std::vector prevSlice(sliceSize, 0); std::vector curSlice(sliceSize, 0); std::vector nextSlice(sliceSize, 0); - // Load initial slices + // Load the first Z-slice (z=0) into curSlice featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + // Pre-load the second Z-slice into nextSlice (if available) if(zPoints > 1) { featureIdsStore.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); @@ -93,8 +163,10 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() { for(int64 x = 0; x < xPoints; x++) { + // Flat index within the current Z-slice buffer const usize inSlice = static_cast(y) * static_cast(xPoints) + static_cast(x); int32 currentFeatureId = curSlice[inSlice]; + // Skip background voxels (FeatureId <= 0) if(currentFeatureId < 1) { continue; @@ -102,7 +174,8 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() float32 onSurface = 0.0f; - // -Z neighbor (index 0): check prevSlice at same (y,x) + // -Z neighbor: same (y, x) position in the previous Z-slice buffer. + // Only valid if z > 0 (not on the bottom face of the volume). if(z > 0) { if(prevSlice[inSlice] != currentFeatureId) @@ -111,7 +184,8 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() } } - // +Z neighbor (index 5): check nextSlice at same (y,x) + // +Z neighbor: same (y, x) position in the next Z-slice buffer. + // Only valid if z < zPoints-1 (not on the top face of the volume). if(z < zPoints - 1) { if(nextSlice[inSlice] != currentFeatureId) @@ -120,7 +194,7 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() } } - // -Y neighbor (index 1): check curSlice at (y-1, x) + // -Y neighbor: one row back in the current slice (inSlice - xPoints). if(y > 0) { if(curSlice[inSlice - static_cast(xPoints)] != currentFeatureId) @@ -129,7 +203,7 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() } } - // +Y neighbor (index 4): check curSlice at (y+1, x) + // +Y neighbor: one row forward in the current slice (inSlice + xPoints). if(y < yPoints - 1) { if(curSlice[inSlice + static_cast(xPoints)] != currentFeatureId) @@ -138,7 +212,7 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() } } - // -X neighbor (index 2): check curSlice at (y, x-1) + // -X neighbor: one element back in the current row (inSlice - 1). if(x > 0) { if(curSlice[inSlice - 1] != currentFeatureId) @@ -147,7 +221,7 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() } } - // +X neighbor (index 3): check curSlice at (y, x+1) + // +X neighbor: one element forward in the current row (inSlice + 1). if(x < xPoints - 1) { if(curSlice[inSlice + 1] != currentFeatureId) @@ -156,25 +230,32 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() } } + // Add this voxel's boundary face contributions to the feature total featureSurfaceArea[currentFeatureId] += onSurface; } } - // Shift the rolling window + // Rotate the rolling window: prevSlice <- curSlice <- nextSlice. + // std::swap is O(1) for vectors (pointer swap, no data copy). std::swap(prevSlice, curSlice); std::swap(curSlice, nextSlice); + // Load the next-next Z-slice into the freed buffer if(z + 2 < zPoints) { featureIdsStore.copyIntoBuffer(static_cast(z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); } } - // Cache the feature-level numCells array locally to avoid per-element OOC lookups + // -- Phase 2: Ratio computation with local caches -- + + // Bulk-read the feature-level NumCells array into a local vector to avoid + // per-element OOC lookups during the ratio computation loop. const usize numFeaturesUSize = static_cast(numFeatures); std::vector localNumCells(numFeaturesUSize); numCells.copyIntoBuffer(0, nonstd::span(localNumCells.data(), numFeaturesUSize)); - // Compute results into a local buffer, then bulk-write to OOC store + // Compute SA/V ratio into a local buffer, then bulk-write to the OOC store. + // This avoids numFeatures individual operator[] writes on the output array. std::vector localSurfaceAreaVolumeRatio(numFeaturesUSize, 0.0f); const float32 thirdRootPi = std::pow(nx::core::Constants::k_PiF, 0.333333f); @@ -183,19 +264,26 @@ Result<> ComputeSurfaceAreaToVolumeScanline::operator()() float32 featureVolume = voxelVol * localNumCells[i]; localSurfaceAreaVolumeRatio[i] = featureSurfaceArea[i] / featureVolume; } + // Single bulk write of the entire SA/V ratio array surfaceAreaVolumeRatio.copyFromBuffer(0, nonstd::span(localSurfaceAreaVolumeRatio.data(), numFeaturesUSize)); + // -- Phase 3: Optional sphericity computation -- + if(m_InputValues->CalculateSphericity) { m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Computing Sphericity")); auto& sphericity = m_DataStructure.getDataAs(m_InputValues->SphericityArrayName)->getDataStoreRef(); + // Compute sphericity into a local buffer, then bulk-write. + // Sphericity = (pi^(1/3) * (6V)^(2/3)) / SA + // A perfect sphere has sphericity = 1.0; irregular shapes have lower values. std::vector localSphericity(numFeaturesUSize, 0.0f); for(usize i = 1; i < numFeaturesUSize; i++) { float32 featureVolume = voxelVol * localNumCells[i]; localSphericity[i] = (thirdRootPi * std::pow((6.0f * featureVolume), 0.66666f)) / featureSurfaceArea[i]; } + // Single bulk write of the entire sphericity array sphericity.copyFromBuffer(0, nonstd::span(localSphericity.data(), numFeaturesUSize)); } diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp index 20b14a2bd8..e648f32d2f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceAreaToVolumeScanline.hpp @@ -11,14 +11,49 @@ struct ComputeSurfaceAreaToVolumeInputValues; /** * @class ComputeSurfaceAreaToVolumeScanline - * @brief Out-of-core algorithm for ComputeSurfaceAreaToVolume. Wraps the voxel iteration in - * chunk-sequential access for guaranteed sequential disk I/O on ZarrStore-backed arrays. - * Feature-level ratio and sphericity computations are unchanged. Selected by DispatchAlgorithm - * when any input array is backed by ZarrStore. + * @brief Out-of-core (OOC) optimized algorithm for computing surface-area-to-volume + * ratio and optional sphericity using Z-slice sequential bulk I/O with a 3-slice + * rolling window. + * + * **The problem this solves**: When the FeatureIds array is stored out-of-core in + * chunked format, the Direct variant's operator[] access to check 6 face neighbors + * triggers chunk thrashing. Each voxel requires reading up to 7 different locations + * (itself + 6 neighbors), and the +/-Z neighbors are dimX*dimY elements apart in + * flat index space, almost certainly spanning different chunks. + * + * **How the rolling window solves it**: This variant reads the FeatureIds array one + * native Z-slice at a time using copyIntoBuffer(), maintaining three in-memory + * buffers (prevSlice, curSlice, nextSlice). All neighbor lookups are performed on + * these buffers: + * - X and Y neighbors: index arithmetic within curSlice (+/-1 and +/-dimX). + * - Z neighbors: same position in prevSlice (-Z) or nextSlice (+Z). + * + * After the voxel scan, the feature-level arrays (NumCells, SA/V ratio, sphericity) + * are also accessed via local std::vector caches with bulk copyIntoBuffer/copyFromBuffer + * calls, avoiding per-element OOC access. + * + * **Surface area accumulation**: Like the Direct variant, this uses a local + * std::vector to accumulate per-feature surface area during the voxel scan. + * Face areas are pre-computed from the image geometry spacing. + * + * **Memory overhead**: 3 input slice buffers (dimX * dimY * 4 bytes each), plus + * local caches for NumCells, SA/V ratio, and sphericity arrays (numFeatures * 4 + * bytes each). For typical datasets, total overhead is a few MB. + * + * @see ComputeSurfaceAreaToVolumeDirect for the in-core variant. + * @see ComputeSurfaceAreaToVolume for the dispatcher. + * @see DispatchAlgorithm for the selection mechanism. */ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeScanline { public: + /** + * @brief Constructs the OOC-optimized SA/V ratio calculator. + * @param dataStructure The DataStructure containing all arrays and the ImageGeom. + * @param mesgHandler Handler for progress/info messages. + * @param shouldCancel Atomic flag for cooperative cancellation. + * @param inputValues Algorithm parameters (geometry path, array paths, sphericity flag). + */ ComputeSurfaceAreaToVolumeScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceAreaToVolumeInputValues* inputValues); ~ComputeSurfaceAreaToVolumeScanline() noexcept; @@ -28,13 +63,22 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceAreaToVolumeScanline ComputeSurfaceAreaToVolumeScanline& operator=(const ComputeSurfaceAreaToVolumeScanline&) = delete; ComputeSurfaceAreaToVolumeScanline& operator=(ComputeSurfaceAreaToVolumeScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized surface-area-to-volume ratio algorithm + * using a 3-slice rolling window with copyIntoBuffer/copyFromBuffer bulk I/O. + * + * All feature-level computations (ratio, sphericity) also use local caches + * with bulk read/write to avoid per-element OOC access. + * + * @return Result<> indicating success or errors. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeSurfaceAreaToVolumeInputValues* m_InputValues = nullptr; ///< Algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cooperative cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message handler. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp index 6997eaeafb..84b4b4db8c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.cpp @@ -8,6 +8,21 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeSurfaceFeatures -- Dispatcher +// +// This file implements the thin dispatch layer for the ComputeSurfaceFeatures +// algorithm. No algorithm logic lives here; the sole responsibility is to +// inspect the storage type of the FeatureIds array and forward execution to +// either ComputeSurfaceFeaturesDirect (in-core) or ComputeSurfaceFeaturesScanline +// (out-of-core), via the DispatchAlgorithm template. +// +// The FeatureIds array is the critical input for dispatch because it is a +// cell-level array (one entry per voxel), which can be very large when stored +// out-of-core. The SurfaceFeatures output is a small feature-level array and +// does not drive the dispatch decision. +// ---------------------------------------------------------------------------- + // ----------------------------------------------------------------------------- ComputeSurfaceFeatures::ComputeSurfaceFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues) @@ -22,6 +37,16 @@ ComputeSurfaceFeatures::ComputeSurfaceFeatures(DataStructure& dataStructure, con ComputeSurfaceFeatures::~ComputeSurfaceFeatures() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Inspects the FeatureIds array's storage type and dispatches to the + * appropriate algorithm variant. + * + * The dispatch decision is made by DispatchAlgorithm, which checks: + * 1. ForceInCoreAlgorithm() -- test override, always selects Direct + * 2. AnyOutOfCore({featureIdsArray}) -- runtime detection of chunked storage + * 3. ForceOocAlgorithm() -- test override, forces Scanline + * 4. Default -- selects Direct (in-core) + */ Result<> ComputeSurfaceFeatures::operator()() { auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp index 0c825a2b1b..e31ad103a8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeatures.hpp @@ -14,23 +14,60 @@ namespace nx::core { +/** + * @struct ComputeSurfaceFeaturesInputValues + * @brief Holds all user-configurable parameters for the ComputeSurfaceFeatures algorithm. + * + * These values are extracted from the filter's parameter map and passed through + * the dispatcher to whichever algorithm variant (Direct or Scanline) is selected. + */ struct SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesInputValues { - AttributeMatrixSelectionParameter::ValueType FeatureAttributeMatrixPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - BoolParameter::ValueType MarkFeature0Neighbors; - DataObjectNameParameter::ValueType SurfaceFeaturesArrayName; + AttributeMatrixSelectionParameter::ValueType FeatureAttributeMatrixPath; ///< Path to the Feature-level AttributeMatrix that sizes the output array. + ArraySelectionParameter::ValueType FeatureIdsPath; ///< Path to the cell-level Int32 FeatureIds array. + GeometrySelectionParameter::ValueType InputImageGeometryPath; ///< Path to the ImageGeom that defines grid dimensions and dimensionality. + BoolParameter::ValueType MarkFeature0Neighbors; ///< When true, features adjacent to FeatureId==0 voxels are marked as surface features. + DataObjectNameParameter::ValueType SurfaceFeaturesArrayName; ///< Name for the output UInt8 surface-features array (created under the FeatureAttributeMatrix). }; /** * @class ComputeSurfaceFeatures - * @brief This algorithm implements support code for the ComputeSurfaceFeaturesFilter + * @brief Dispatcher that selects between the in-core (Direct) and out-of-core (Scanline) + * surface-feature identification algorithms at runtime. + * + * This class contains no algorithm logic. Its operator()() inspects the storage backing + * of the FeatureIds array and calls + * `DispatchAlgorithm(...)`. + * + * **Algorithm overview**: A feature is considered a "surface feature" if any voxel + * belonging to that feature satisfies one of these conditions: + * 1. The voxel sits on the outer boundary of the image geometry (x/y/z min or max). + * 2. The voxel has a face neighbor with FeatureId == 0 (when MarkFeature0Neighbors + * is enabled). + * + * The algorithm supports both 3D geometries (full 6-neighbor check) and 2D geometries + * (one dimension is degenerate, reducing to a 4-neighbor check on the non-degenerate + * plane). + * + * The output is a feature-level UInt8 array where 0 = interior, 1 = surface. + * + * **Dispatch rules** (see AlgorithmDispatch.hpp): + * - If all input arrays are in-memory, the Direct variant is selected. + * - If any input array uses OOC storage, the Scanline variant is selected. + * - Test-override flags can force either path. + * + * @see ComputeSurfaceFeaturesDirect, ComputeSurfaceFeaturesScanline, DispatchAlgorithm */ - class SIMPLNXCORE_EXPORT ComputeSurfaceFeatures { public: + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all arrays and geometries. + * @param mesgHandler Handler for sending progress/info messages back to the UI. + * @param shouldCancel Atomic flag checked periodically to support user cancellation. + * @param inputValues User-configured parameters for the algorithm. + */ ComputeSurfaceFeatures(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ComputeSurfaceFeaturesInputValues* inputValues); ~ComputeSurfaceFeatures() noexcept; @@ -39,13 +76,18 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceFeatures ComputeSurfaceFeatures& operator=(const ComputeSurfaceFeatures&) = delete; ComputeSurfaceFeatures& operator=(ComputeSurfaceFeatures&&) noexcept = delete; + /** + * @brief Dispatches to the appropriate algorithm variant (Direct or Scanline) + * based on whether the FeatureIds array uses out-of-core storage. + * @return Result<> indicating success or any errors encountered. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; ///< User-configured algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Atomic flag for cooperative cancellation. + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress and informational messages. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp index 7dd9731578..7ab20ba8b3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.cpp @@ -8,12 +8,51 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeSurfaceFeaturesDirect -- In-Core Algorithm +// +// Identifies which features touch the outer surface of the image geometry or +// border FeatureId==0 voxels. The output is a feature-level boolean (UInt8) +// array: 0 = interior feature, 1 = surface feature. +// +// Data access pattern: Uses operator[] on the FeatureIds DataStore for both +// the voxel under test and its face neighbors. This is efficient when the +// DataStore is contiguous in-memory, but would cause chunk thrashing on OOC +// storage because neighbor lookups span large index offsets (+/-dimX*dimY +// for Z neighbors). +// +// Two helper functions handle the dimensionality-specific logic: +// - findSurfaceFeatures3D: Full 6-neighbor check for 3D geometries. +// - findSurfaceFeatures2D: 4-neighbor check on the non-degenerate plane +// for geometries where one dimension has size 1. +// +// Within each helper, two private IsPointASurfaceFeature overloads (2D/3D) +// encapsulate the boundary and neighbor-zero checks for a single voxel. +// ---------------------------------------------------------------------------- + namespace { +/** + * @brief Checks whether a single voxel in a 2D geometry qualifies its owning + * feature as a surface feature. + * + * A voxel is on the surface if: + * - It sits on the outer boundary of the 2D plane (x or y == 0 or max). + * - Any of its 4 face neighbors has FeatureId == 0 (when markFeature0Neighbors + * is true). + * + * @param point The (x, y) coordinates of the voxel in the 2D plane. + * @param xPoints Number of cells in the remapped X dimension. + * @param yPoints Number of cells in the remapped Y dimension. + * @param markFeature0Neighbors Whether to check neighbors for FeatureId == 0. + * @param featureIds The cell-level FeatureIds DataStore (accessed via operator[]). + * @return true if the voxel makes its feature a surface feature. + */ bool IsPointASurfaceFeature(const Point2D& point, usize xPoints, usize yPoints, bool markFeature0Neighbors, const Int32AbstractDataStore& featureIds) { const usize yStride = point.getY() * xPoints; + // Boundary check: voxels on the outer edge of the plane are always surface voxels if(point.getX() <= 0 || point.getX() >= xPoints - 1) { return true; @@ -23,20 +62,26 @@ bool IsPointASurfaceFeature(const Point2D& point, usize xPoints, usize yP return true; } + // Neighbor-zero check: if any face neighbor has FeatureId == 0, the feature + // is considered to touch the surface (feature 0 typically represents empty space) if(markFeature0Neighbors) { + // -X neighbor if(featureIds[yStride + point.getX() - 1] == 0) { return true; } + // +X neighbor if(featureIds[yStride + point.getX() + 1] == 0) { return true; } + // -Y neighbor if(featureIds[yStride + point.getX() - xPoints] == 0) { return true; } + // +Y neighbor if(featureIds[yStride + point.getX() + xPoints] == 0) { return true; @@ -46,11 +91,29 @@ bool IsPointASurfaceFeature(const Point2D& point, usize xPoints, usize yP return false; } +/** + * @brief Checks whether a single voxel in a 3D geometry qualifies its owning + * feature as a surface feature. + * + * A voxel is on the surface if: + * - It sits on the outer boundary of the volume (x, y, or z == 0 or max). + * - Any of its 6 face neighbors has FeatureId == 0 (when markFeature0Neighbors + * is true). + * + * @param point The (x, y, z) coordinates of the voxel. + * @param xPoints Number of cells in X. + * @param yPoints Number of cells in Y. + * @param zPoints Number of cells in Z. + * @param markFeature0Neighbors Whether to check neighbors for FeatureId == 0. + * @param featureIds The cell-level FeatureIds DataStore (accessed via operator[]). + * @return true if the voxel makes its feature a surface feature. + */ bool IsPointASurfaceFeature(const Point3D& point, usize xPoints, usize yPoints, usize zPoints, bool markFeature0Neighbors, const Int32AbstractDataStore& featureIds) { usize yStride = point.getY() * xPoints; usize zStride = point.getZ() * xPoints * yPoints; + // Boundary check: voxels on the outer faces of the volume are always surface voxels if(point.getX() <= 0 || point.getX() >= xPoints - 1) { return true; @@ -64,28 +127,35 @@ bool IsPointASurfaceFeature(const Point3D& point, usize xPoints, usize yP return true; } + // Neighbor-zero check: test all 6 face neighbors for FeatureId == 0 if(markFeature0Neighbors) { + // -X neighbor if(featureIds[zStride + yStride + point.getX() - 1] == 0) { return true; } + // +X neighbor if(featureIds[zStride + yStride + point.getX() + 1] == 0) { return true; } + // -Y neighbor (one row back = -xPoints in flat index) if(featureIds[zStride + yStride + point.getX() - xPoints] == 0) { return true; } + // +Y neighbor (one row forward = +xPoints in flat index) if(featureIds[zStride + yStride + point.getX() + xPoints] == 0) { return true; } + // -Z neighbor (one slice back = -(xPoints*yPoints) in flat index) if(featureIds[zStride + yStride + point.getX() - (xPoints * yPoints)] == 0) { return true; } + // +Z neighbor (one slice forward = +(xPoints*yPoints) in flat index) if(featureIds[zStride + yStride + point.getX() + (xPoints * yPoints)] == 0) { return true; @@ -95,6 +165,22 @@ bool IsPointASurfaceFeature(const Point3D& point, usize xPoints, usize yP return false; } +/** + * @brief Identifies surface features in a 3D image geometry using direct array access. + * + * Iterates all voxels in Z-Y-X order. For each voxel with a non-zero FeatureId + * whose feature has not already been marked, calls IsPointASurfaceFeature to + * check boundary and neighbor-zero conditions. Once a feature is marked as + * surface (surfaceFeatures[gNum] = 1), subsequent voxels of that feature are + * skipped (short-circuit optimization). + * + * @param dataStructure The DataStructure containing all arrays. + * @param featureGeometryPathValue Path to the ImageGeom. + * @param featureIdsArrayPathValue Path to the FeatureIds cell array. + * @param surfaceFeaturesArrayPathValue Path to the output SurfaceFeatures feature array. + * @param markFeature0Neighbors Whether to treat FeatureId==0 neighbors as surface indicators. + * @param shouldCancel Cancellation flag checked once per Z-slice. + */ void findSurfaceFeatures3D(DataStructure& dataStructure, const DataPath& featureGeometryPathValue, const DataPath& featureIdsArrayPathValue, const DataPath& surfaceFeaturesArrayPathValue, bool markFeature0Neighbors, const std::atomic_bool& shouldCancel) { @@ -120,6 +206,7 @@ void findSurfaceFeatures3D(DataStructure& dataStructure, const DataPath& feature for(usize x = 0; x < xPoints; x++) { const int32 gNum = featureIds[zStride + yStride + x]; + // Skip feature 0 (background) and features already marked as surface if(gNum != 0 && !surfaceFeatures[gNum]) { if(IsPointASurfaceFeature(Point3D{x, y, z}, xPoints, yPoints, zPoints, markFeature0Neighbors, featureIds)) @@ -132,6 +219,21 @@ void findSurfaceFeatures3D(DataStructure& dataStructure, const DataPath& feature } } +/** + * @brief Identifies surface features in a 2D image geometry using direct array access. + * + * Determines which dimension is degenerate (size == 1) and remaps the remaining + * two dimensions to a 2D (xPoints, yPoints) coordinate system. Then iterates + * all voxels in the 2D plane, checking boundary and neighbor-zero conditions + * with IsPointASurfaceFeature (2D overload). + * + * @param dataStructure The DataStructure containing all arrays. + * @param featureGeometryPathValue Path to the ImageGeom. + * @param featureIdsArrayPathValue Path to the FeatureIds cell array. + * @param surfaceFeaturesArrayPathValue Path to the output SurfaceFeatures feature array. + * @param markFeature0Neighbors Whether to treat FeatureId==0 neighbors as surface indicators. + * @param shouldCancel Cancellation flag checked once per Y-row. + */ void findSurfaceFeatures2D(DataStructure& dataStructure, const DataPath& featureGeometryPathValue, const DataPath& featureIdsArrayPathValue, const DataPath& surfaceFeaturesArrayPathValue, bool markFeature0Neighbors, const std::atomic_bool& shouldCancel) { @@ -139,6 +241,9 @@ void findSurfaceFeatures2D(DataStructure& dataStructure, const DataPath& feature const auto& featureIds = dataStructure.getDataAs(featureIdsArrayPathValue)->getDataStoreRef(); auto& surfaceFeatures = dataStructure.getDataAs(surfaceFeaturesArrayPathValue)->getDataStoreRef(); + // Determine which two dimensions form the non-degenerate plane. + // The degenerate dimension (size == 1) is collapsed, and the remaining + // two dimensions are remapped to xPoints and yPoints for the 2D algorithm. usize xPoints = 0; usize yPoints = 0; @@ -169,6 +274,7 @@ void findSurfaceFeatures2D(DataStructure& dataStructure, const DataPath& feature for(usize x = 0; x < xPoints; x++) { const int32 gNum = featureIds[yStride + x]; + // Skip feature 0 (background) and features already marked if(gNum != 0 && surfaceFeatures[gNum] == 0) { if(IsPointASurfaceFeature(Point2D{x, y}, xPoints, yPoints, markFeature0Neighbors, featureIds)) @@ -196,26 +302,39 @@ ComputeSurfaceFeaturesDirect::~ComputeSurfaceFeaturesDirect() noexcept = default // ----------------------------------------------------------------------------- /** - * @brief Identifies surface features using direct Z-Y-X iteration. - * In-core path: delegates to findSurfaceFeatures3D or findSurfaceFeatures2D - * depending on geometry dimensionality. + * @brief Identifies surface features using direct in-memory array indexing. + * + * The algorithm proceeds as follows: + * 1. Build the output array path from the FeatureAttributeMatrix and output name. + * 2. Validate that the FeatureIds values are consistent with the AttributeMatrix + * tuple count (catches mismatches that would cause out-of-bounds writes). + * 3. Query the image geometry dimensionality: + * - 3D: delegate to findSurfaceFeatures3D (6-neighbor check). + * - 2D: delegate to findSurfaceFeatures2D (4-neighbor check on the + * non-degenerate plane). + * - Other: return an error. + * + * @return Result<> indicating success, validation errors, or unsupported geometry. */ Result<> ComputeSurfaceFeaturesDirect::operator()() { + // Extract input values into local variables for readability const auto pMarkFeature0NeighborsValue = m_InputValues->MarkFeature0Neighbors; const auto pFeatureGeometryPathValue = m_InputValues->InputImageGeometryPath; const auto pFeatureIdsArrayPathValue = m_InputValues->FeatureIdsPath; const auto pFeaturesAttributeMatrixPathValue = m_InputValues->FeatureAttributeMatrixPath; + // The output SurfaceFeatures array lives under the Feature AttributeMatrix const auto pSurfaceFeaturesArrayPathValue = pFeaturesAttributeMatrixPathValue.createChildPath(m_InputValues->SurfaceFeaturesArrayName); + // Validate that the max FeatureId does not exceed the AttributeMatrix size const auto& featureIdsArray = m_DataStructure.getDataRefAs(pFeatureIdsArrayPathValue); - auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, pFeaturesAttributeMatrixPathValue, featureIdsArray, false, m_MessageHandler); if(validateNumFeatResult.invalid()) { return validateNumFeatResult; } + // Branch on geometry dimensionality const auto& featureGeometry = m_DataStructure.getDataRefAs(pFeatureGeometryPathValue); if(const usize geometryDimensionality = featureGeometry.getDimensionality(); geometryDimensionality == 3) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp index ebacffdb14..5df9b65423 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesDirect.hpp @@ -11,13 +11,44 @@ struct ComputeSurfaceFeaturesInputValues; /** * @class ComputeSurfaceFeaturesDirect - * @brief In-core algorithm for ComputeSurfaceFeatures. Preserves the original 2D/3D branching - * with sequential voxel iteration and face-neighbor surface detection. Selected by - * DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * @brief In-core (direct memory access) algorithm for identifying surface features. + * + * This is the traditional algorithm that uses operator[] to read FeatureIds and write + * SurfaceFeatures directly through the DataStore abstraction. It supports both 3D and + * 2D image geometries, branching into separate helper functions based on the geometry's + * dimensionality. + * + * **When this variant is selected**: DispatchAlgorithm selects this class when all + * input arrays are backed by contiguous in-memory DataStore. + * + * **3D algorithm**: Iterates all voxels in Z-Y-X order. For each voxel, checks: + * - Whether the voxel is on the outer boundary (x/y/z == 0 or max). + * - Whether any of its 6 face neighbors has FeatureId == 0 (if MarkFeature0Neighbors + * is enabled). + * If either condition is met, the feature owning that voxel is marked as a surface feature. + * + * **2D algorithm**: Determines which dimension is degenerate (size == 1) and performs + * the equivalent 4-neighbor boundary check on the non-degenerate plane. + * + * **Why a separate OOC variant exists**: The Direct variant accesses the FeatureIds + * array via operator[], and for the 3D case, neighbor lookups span +/-1, +/-dimX, + * and +/-(dimX*dimY) in flat index space. When FeatureIds is stored in chunked OOC + * format, these scattered accesses cause chunk thrashing. The Scanline variant reads + * entire Z-slices sequentially to avoid this. + * + * @see ComputeSurfaceFeaturesScanline for the OOC-optimized variant. + * @see ComputeSurfaceFeatures for the dispatcher. */ class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesDirect { public: + /** + * @brief Constructs the in-core surface feature identifier. + * @param dataStructure The DataStructure containing FeatureIds and SurfaceFeatures arrays. + * @param mesgHandler Handler for progress/info messages. + * @param shouldCancel Atomic flag for cooperative cancellation. + * @param inputValues Algorithm parameters (geometry path, array paths, flags). + */ ComputeSurfaceFeaturesDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceFeaturesInputValues* inputValues); ~ComputeSurfaceFeaturesDirect() noexcept; @@ -26,13 +57,21 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesDirect ComputeSurfaceFeaturesDirect& operator=(const ComputeSurfaceFeaturesDirect&) = delete; ComputeSurfaceFeaturesDirect& operator=(ComputeSurfaceFeaturesDirect&&) noexcept = delete; + /** + * @brief Executes the in-core surface feature identification algorithm. + * + * Validates the feature-to-attribute-matrix mapping, determines whether the + * geometry is 2D or 3D, and delegates to the appropriate helper function. + * + * @return Result<> indicating success, errors, or unsupported dimensionality. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; ///< Algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cooperative cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message handler. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp index 88a852252f..78cbc84091 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.cpp @@ -12,23 +12,73 @@ using namespace nx::core; +// ---------------------------------------------------------------------------- +// ComputeSurfaceFeaturesScanline -- Out-of-Core Algorithm +// +// Identifies which features touch the outer surface of the image geometry or +// border FeatureId==0 voxels. Produces the same output as the Direct variant: +// a feature-level UInt8 array where 0 = interior, 1 = surface. +// +// KEY DESIGN PRINCIPLE: All access to the large cell-level FeatureIds array is +// strictly sequential by Z-slice, using copyIntoBuffer() for bulk reads. The +// small feature-level SurfaceFeatures array is cached locally in a std::vector +// and written back in a single copyFromBuffer() call at the end. +// +// ROLLING WINDOW: Three std::vector buffers (prevSlice, curSlice, +// nextSlice) hold adjacent Z-slices so that all neighbor lookups are simple +// in-memory array accesses with no disk I/O. +// +// 2D GEOMETRY HANDLING: Unlike the Direct variant which has separate 2D/3D +// code paths, the Scanline variant always iterates the native Z-Y-X grid and +// remaps coordinates to the 2D plane as needed. This unified approach +// maintains sequential Z-slice I/O even for 2D geometries where the +// degenerate dimension is X or Y (not Z). +// +// For the degenerate-Z case (zPoints==1), all data fits in a single Z-slice, +// so prevSlice/nextSlice are unused and all 4 neighbors come from curSlice. +// For degenerate-X or degenerate-Y cases, the remapped-Y direction maps to +// the native Z axis, so the +/-Y neighbors come from prevSlice/nextSlice. +// ---------------------------------------------------------------------------- + namespace { /** - * @brief Checks whether a voxel at (remappedX, remappedY) in a 2D geometry is - * on the surface. Operates on a pair of flat buffers that hold the current - * remapped-Y row and its neighbours in the remapped-Y direction. + * @brief Checks whether a voxel in a 2D geometry qualifies its feature as a + * surface feature, using the rolling-window slice buffers. + * + * This function handles coordinate remapping from the native 3D Z-Y-X grid + * to the logical 2D plane. The remapping depends on which dimension is + * degenerate (size == 1): + * + * - **Degenerate Z (zPoints==1)**: The entire dataset is a single Z-slice. + * remappedX = native X, remappedY = native Y. All 4 neighbors live in + * curSlice using standard Y*xPoints+X indexing. * - * For the degenerate-Z case (zPoints==1) the entire dataset is one Z-slice, - * so prevRow/nextRow are unused for Z neighbours -- all four 2D neighbours - * live in curSlice. + * - **Degenerate X or Y**: The non-degenerate in-plane dimension maps to + * remappedX (contiguous in memory), and the native Z dimension maps to + * remappedY. The +/-remappedX neighbors are at nativeInSlice +/- 1 in + * curSlice. The +/-remappedY neighbors come from prevSlice/nextSlice + * (the adjacent native Z-slices). * - * For the degenerate-X or degenerate-Y cases, the remapped-Y direction maps - * to the native Z axis, so prevRow/nextRow come from prevSlice/nextSlice. + * @param remappedX X coordinate in the logical 2D plane. + * @param remappedY Y coordinate in the logical 2D plane. + * @param remappedXPoints Number of cells in the remapped X dimension. + * @param remappedYPoints Number of cells in the remapped Y dimension. + * @param markFeature0Neighbors Whether to check for FeatureId==0 neighbors. + * @param curSlice Buffer holding the current native Z-slice's FeatureIds. + * @param prevSlice Buffer holding the previous native Z-slice's FeatureIds. + * @param nextSlice Buffer holding the next native Z-slice's FeatureIds. + * @param nativeInSlice Flat index of this voxel within the native Z-slice + * (y * nativeXPoints + x). Used for non-degenerate-Z neighbor lookups. + * @param hasPrevSlice True if a previous Z-slice exists (z > 0). + * @param hasNextSlice True if a next Z-slice exists (z + 1 < zPoints). + * @param degenerateZ True if the Z dimension has size 1. + * @return true if the voxel makes its feature a surface feature. */ bool IsPointASurfaceFeature2D(usize remappedX, usize remappedY, usize remappedXPoints, usize remappedYPoints, bool markFeature0Neighbors, const std::vector& curSlice, const std::vector& prevSlice, const std::vector& nextSlice, usize nativeInSlice, bool hasPrevSlice, bool hasNextSlice, bool degenerateZ) { + // Boundary check: voxels on the outer edges of the 2D plane are surface voxels if(remappedX <= 0 || remappedX >= remappedXPoints - 1) { return true; @@ -42,21 +92,26 @@ bool IsPointASurfaceFeature2D(usize remappedX, usize remappedY, usize remappedXP { if(degenerateZ) { - // All 4 neighbours are in curSlice (the single Z-slice holds everything) - // For degenerate Z: remapped coords map directly to native Y*xPoints+X layout + // DEGENERATE Z: All data is in one Z-slice. The 4 neighbors in the + // 2D plane are all within curSlice using the native Y*xPoints+X layout + // (which matches the remapped layout since remappedX=X, remappedY=Y). const usize yStride = remappedY * remappedXPoints; + // -X neighbor if(curSlice[yStride + remappedX - 1] == 0) { return true; } + // +X neighbor if(curSlice[yStride + remappedX + 1] == 0) { return true; } + // -Y neighbor (previous row in the same slice) if(curSlice[(remappedY - 1) * remappedXPoints + remappedX] == 0) { return true; } + // +Y neighbor (next row in the same slice) if(curSlice[(remappedY + 1) * remappedXPoints + remappedX] == 0) { return true; @@ -64,28 +119,29 @@ bool IsPointASurfaceFeature2D(usize remappedX, usize remappedY, usize remappedXP } else { - // Remapped Y direction maps to native Z, so +/-Y neighbours come - // from prevSlice / nextSlice. Remapped X neighbours are within - // curSlice at stride ±1 from the native in-slice index (works for - // both degenerate-X and degenerate-Y because the non-degenerate - // in-plane dimension is always contiguous in memory). + // DEGENERATE X or Y: The remapped Y direction maps to the native Z + // axis, so +/-remappedY neighbors come from the adjacent Z-slices + // (prevSlice/nextSlice). The remapped X neighbors are within curSlice + // at +/-1 from the native in-slice index. This works for both + // degenerate-X and degenerate-Y because the non-degenerate in-plane + // dimension is always contiguous in the native Z-slice layout. - // -remappedX neighbour + // -remappedX neighbor (adjacent element in the current Z-slice) if(curSlice[nativeInSlice - 1] == 0) { return true; } - // +remappedX neighbour + // +remappedX neighbor if(curSlice[nativeInSlice + 1] == 0) { return true; } - // -remappedY neighbour (previous Z-slice) + // -remappedY neighbor (same position in the previous native Z-slice) if(hasPrevSlice && prevSlice[nativeInSlice] == 0) { return true; } - // +remappedY neighbour (next Z-slice) + // +remappedY neighbor (same position in the next native Z-slice) if(hasNextSlice && nextSlice[nativeInSlice] == 0) { return true; @@ -97,12 +153,34 @@ bool IsPointASurfaceFeature2D(usize remappedX, usize remappedY, usize remappedXP } /** - * @brief Checks whether a voxel at (x,y) within a Z-slice is on the surface - * in 3D. Uses the three-slice rolling window: prevSlice, curSlice, nextSlice. + * @brief Checks whether a voxel in a 3D geometry qualifies its feature as a + * surface feature, using the rolling-window slice buffers. + * + * All 6 face-neighbor lookups use the in-memory buffers: + * - +/-X: curSlice at inSlice +/- 1 + * - +/-Y: curSlice at inSlice +/- xPoints (one row offset) + * - -Z: prevSlice at the same inSlice position + * - +Z: nextSlice at the same inSlice position + * + * This avoids any direct access to the OOC DataStore, which is the entire + * point of the Scanline approach. + * + * @param x X coordinate of the voxel. + * @param y Y coordinate of the voxel. + * @param z Z coordinate of the voxel. + * @param xPoints Number of cells in X. + * @param yPoints Number of cells in Y. + * @param zPoints Number of cells in Z. + * @param markFeature0Neighbors Whether to check for FeatureId==0 neighbors. + * @param prevSlice Buffer holding Z-slice (z-1). + * @param curSlice Buffer holding Z-slice (z). + * @param nextSlice Buffer holding Z-slice (z+1). + * @return true if the voxel makes its feature a surface feature. */ bool IsPointASurfaceFeature3D(usize x, usize y, usize z, usize xPoints, usize yPoints, usize zPoints, bool markFeature0Neighbors, const std::vector& prevSlice, const std::vector& curSlice, const std::vector& nextSlice) { + // Boundary check: voxels on the outer faces of the volume are surface voxels if(x <= 0 || x >= xPoints - 1) { return true; @@ -116,36 +194,38 @@ bool IsPointASurfaceFeature3D(usize x, usize y, usize z, usize xPoints, usize yP return true; } + // Neighbor-zero check: test all 6 face neighbors for FeatureId == 0 if(markFeature0Neighbors) { + // Compute the flat index within the Z-slice buffer const usize inSlice = y * xPoints + x; - // -X + // -X neighbor (one element back in the current row) if(curSlice[inSlice - 1] == 0) { return true; } - // +X + // +X neighbor (one element forward in the current row) if(curSlice[inSlice + 1] == 0) { return true; } - // -Y + // -Y neighbor (one row back = -xPoints elements) if(curSlice[inSlice - xPoints] == 0) { return true; } - // +Y + // +Y neighbor (one row forward = +xPoints elements) if(curSlice[inSlice + xPoints] == 0) { return true; } - // -Z (previous slice) + // -Z neighbor (same position in the previous Z-slice buffer) if(prevSlice[inSlice] == 0) { return true; } - // +Z (next slice) + // +Z neighbor (same position in the next Z-slice buffer) if(nextSlice[inSlice] == 0) { return true; @@ -171,21 +251,43 @@ ComputeSurfaceFeaturesScanline::~ComputeSurfaceFeaturesScanline() noexcept = def // ----------------------------------------------------------------------------- /** - * @brief Identifies surface features using Z-slice bulk I/O with a rolling window. - * OOC path: reads featureIds one Z-slice at a time via copyIntoBuffer(), - * keeping 3 slices resident (prev/cur/next) for cross-slice neighbour access. - * Handles both 3D and 2D geometries with coordinate remapping. + * @brief Identifies surface features using a 3-slice rolling window with sequential + * bulk I/O for out-of-core storage compatibility. + * + * The algorithm has four phases: + * + * **Phase 1 -- Validation and setup**: Validate FeatureId-to-AttributeMatrix + * indexing, cache the small feature-level SurfaceFeatures array locally, and + * compute remapped dimensions for 2D geometries. + * + * **Phase 2 -- Rolling window initialization**: Allocate three Z-slice buffers + * and load the first one or two slices from the FeatureIds OOC store. + * + * **Phase 3 -- Z-slice iteration**: For each native Z-slice: + * - Iterate all voxels in Y-X order within curSlice. + * - For 3D geometries: call IsPointASurfaceFeature3D with the three buffers. + * - For 2D geometries: remap native (x, y, z) to the logical 2D plane and + * call IsPointASurfaceFeature2D. + * - Rotate the rolling window and load the next Z-slice. + * + * **Phase 4 -- Write-back**: Bulk-write the local SurfaceFeatures vector back + * to the OOC store in a single copyFromBuffer() call. + * + * @return Result<> indicating success, validation errors, or unsupported dimensionality. */ Result<> ComputeSurfaceFeaturesScanline::operator()() { + // -- Phase 1: Validation and setup -- + + // Extract input values into local variables for readability const auto pMarkFeature0NeighborsValue = m_InputValues->MarkFeature0Neighbors; const auto pFeatureGeometryPathValue = m_InputValues->InputImageGeometryPath; const auto pFeatureIdsArrayPathValue = m_InputValues->FeatureIdsPath; const auto pFeaturesAttributeMatrixPathValue = m_InputValues->FeatureAttributeMatrixPath; const auto pSurfaceFeaturesArrayPathValue = pFeaturesAttributeMatrixPathValue.createChildPath(m_InputValues->SurfaceFeaturesArrayName); + // Validate that the max FeatureId does not exceed the AttributeMatrix size const auto& featureIdsArray = m_DataStructure.getDataRefAs(pFeatureIdsArrayPathValue); - auto validateNumFeatResult = ValidateFeatureIdsToFeatureAttributeMatrixIndexing(m_DataStructure, pFeaturesAttributeMatrixPathValue, featureIdsArray, false, m_MessageHandler); if(validateNumFeatResult.invalid()) { @@ -196,8 +298,12 @@ Result<> ComputeSurfaceFeaturesScanline::operator()() auto& featureIds = m_DataStructure.getDataAs(pFeatureIdsArrayPathValue)->getDataStoreRef(); auto& surfaceFeatures = m_DataStructure.getDataAs(pSurfaceFeaturesArrayPathValue)->getDataStoreRef(); - // Cache the small feature-level surfaceFeatures array locally to avoid - // per-voxel OOC operator[] in the hot Z-Y-X loop. + // Cache the small feature-level SurfaceFeatures array in a local vector. + // This is critical because the inner Z-Y-X loop indexes into this array + // by FeatureId (e.g., localSurfaceFeatures[gNum]). If the SurfaceFeatures + // array were OOC, each of these lookups would be a random-access read/write + // to a chunked store. By caching locally, we keep all feature-level access + // in fast contiguous memory. const usize numFeatures = surfaceFeatures.getNumberOfTuples(); std::vector localSurfaceFeatures(numFeatures, 0); surfaceFeatures.copyIntoBuffer(0, nonstd::span(localSurfaceFeatures.data(), numFeatures)); @@ -207,7 +313,13 @@ Result<> ComputeSurfaceFeaturesScanline::operator()() const usize zPoints = featureGeometry.getNumZCells(); const usize geometryDimensionality = featureGeometry.getDimensionality(); - // For 2D geometries, compute the remapped dimensions + // For 2D geometries, determine the remapped dimensions. + // The degenerate dimension (size == 1) is collapsed, and the remaining two + // dimensions become remappedXPoints and remappedYPoints. The remapping + // determines how native (x, y, z) coordinates map to the 2D plane: + // - Degenerate X (xPoints==1): remappedX = native Y, remappedY = native Z + // - Degenerate Y (yPoints==1): remappedX = native X, remappedY = native Z + // - Degenerate Z (zPoints==1): remappedX = native X, remappedY = native Y usize remappedXPoints = 0; usize remappedYPoints = 0; if(geometryDimensionality == 2) @@ -229,19 +341,25 @@ Result<> ComputeSurfaceFeaturesScanline::operator()() } } - // Z-slice rolling window for bulk I/O + // -- Phase 2: Rolling window initialization -- + + // Each native Z-slice has yPoints * xPoints voxels. This is the granularity + // of bulk I/O -- one copyIntoBuffer() call per Z-slice. const usize sliceSize = yPoints * xPoints; std::vector prevSlice(sliceSize, 0); std::vector curSlice(sliceSize, 0); std::vector nextSlice(sliceSize, 0); - // Load initial slices + // Load the first native Z-slice into curSlice featureIds.copyIntoBuffer(0, nonstd::span(curSlice.data(), sliceSize)); + // Pre-load the second Z-slice if available if(zPoints > 1) { featureIds.copyIntoBuffer(sliceSize, nonstd::span(nextSlice.data(), sliceSize)); } + // -- Phase 3: Z-slice iteration with rolling window -- + for(usize z = 0; z < zPoints; z++) { if(m_ShouldCancel) @@ -252,12 +370,19 @@ Result<> ComputeSurfaceFeaturesScanline::operator()() { for(usize x = 0; x < xPoints; x++) { + // Compute the flat index within the current Z-slice buffer const usize inSlice = y * xPoints + x; const int32 gNum = curSlice[inSlice]; + + // Skip feature 0 (background) and features already marked as surface. + // The short-circuit on localSurfaceFeatures[gNum] avoids redundant + // neighbor checks for features that have already been identified. if(gNum != 0 && !localSurfaceFeatures[gNum]) { if(geometryDimensionality == 3) { + // 3D: Check boundary position and 6 face neighbors via the + // three rolling-window buffers if(IsPointASurfaceFeature3D(x, y, z, xPoints, yPoints, zPoints, pMarkFeature0NeighborsValue, prevSlice, curSlice, nextSlice)) { localSurfaceFeatures[gNum] = 1; @@ -265,22 +390,26 @@ Result<> ComputeSurfaceFeaturesScanline::operator()() } else if(geometryDimensionality == 2) { - // Remap native 3D coordinates to 2D based on degenerate dimension + // 2D: Remap native 3D coordinates (x, y, z) to the logical 2D + // plane based on which dimension is degenerate. usize remappedX = 0; usize remappedY = 0; bool degenerateZ = false; if(xPoints == 1) { + // Degenerate X: the YZ plane is the 2D plane remappedX = y; remappedY = z; } else if(yPoints == 1) { + // Degenerate Y: the XZ plane is the 2D plane remappedX = x; remappedY = z; } else // zPoints == 1 { + // Degenerate Z: the XY plane is the 2D plane (most common case) remappedX = x; remappedY = y; degenerateZ = true; @@ -300,16 +429,20 @@ Result<> ComputeSurfaceFeaturesScanline::operator()() } } - // Shift the rolling window + // Rotate the rolling window: prevSlice <- curSlice <- nextSlice. + // std::swap is O(1) for vectors (pointer swap only, no data copy). std::swap(prevSlice, curSlice); std::swap(curSlice, nextSlice); + // Load the next-next Z-slice into the freed buffer if(z + 2 < zPoints) { featureIds.copyIntoBuffer((z + 2) * sliceSize, nonstd::span(nextSlice.data(), sliceSize)); } } - // Write cached results back to the OOC store + // -- Phase 4: Write-back -- + // Bulk-write the locally cached SurfaceFeatures results back to the OOC + // store. This is a single sequential write of the entire feature-level array. surfaceFeatures.copyFromBuffer(0, nonstd::span(localSurfaceFeatures.data(), numFeatures)); return {}; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp index 3e757e7c01..bf3ffaa0f8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeSurfaceFeaturesScanline.hpp @@ -11,14 +11,50 @@ struct ComputeSurfaceFeaturesInputValues; /** * @class ComputeSurfaceFeaturesScanline - * @brief Out-of-core algorithm for ComputeSurfaceFeatures. Uses chunk-sequential 3D iteration - * with 2D coordinate remapping for degenerate dimensions, ensuring sequential disk I/O - * on ZarrStore-backed arrays. Selected by DispatchAlgorithm when any input array is - * backed by ZarrStore. + * @brief Out-of-core (OOC) optimized algorithm for identifying surface features using + * Z-slice sequential bulk I/O with a 3-slice rolling window. + * + * **The problem this solves**: When the FeatureIds array is stored out-of-core in + * chunked format, the Direct variant's operator[] access to check face neighbors + * triggers chunk thrashing -- especially the +/-Z neighbor lookups that are + * dimX*dimY elements apart in flat index space. This makes the algorithm orders + * of magnitude slower on OOC data. + * + * **How the rolling window solves it**: This variant reads the FeatureIds array one + * native Z-slice at a time using copyIntoBuffer(), maintaining three in-memory + * buffers (prevSlice, curSlice, nextSlice). All neighbor lookups are performed on + * these in-memory buffers: + * - X and Y neighbors: simple index arithmetic within curSlice. + * - Z neighbors: same position in prevSlice (-Z) or nextSlice (+Z). + * + * **2D geometry support**: For geometries with one degenerate dimension (size == 1), + * the algorithm still iterates the native Z-Y-X grid but remaps coordinates to + * the 2D plane for boundary and neighbor checks. This unified approach avoids + * separate 2D/3D code paths while maintaining sequential I/O. + * + * **Output caching**: The SurfaceFeatures output is a small feature-level array + * (one element per feature, not per voxel). To avoid per-voxel OOC writes, the + * results are accumulated in a local std::vector and bulk-written once at the end + * via copyFromBuffer(). + * + * **Memory overhead**: 3 input buffers of size (dimX * dimY * 4 bytes) for the + * rolling window, plus a local vector of size (numFeatures * 1 byte) for the + * cached output. For typical datasets this is a few MB. + * + * @see ComputeSurfaceFeaturesDirect for the in-core variant. + * @see ComputeSurfaceFeatures for the dispatcher. + * @see DispatchAlgorithm for the selection mechanism. */ class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesScanline { public: + /** + * @brief Constructs the OOC-optimized surface feature identifier. + * @param dataStructure The DataStructure containing FeatureIds and SurfaceFeatures arrays. + * @param mesgHandler Handler for progress/info messages. + * @param shouldCancel Atomic flag for cooperative cancellation. + * @param inputValues Algorithm parameters (geometry path, array paths, flags). + */ ComputeSurfaceFeaturesScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const ComputeSurfaceFeaturesInputValues* inputValues); ~ComputeSurfaceFeaturesScanline() noexcept; @@ -27,13 +63,22 @@ class SIMPLNXCORE_EXPORT ComputeSurfaceFeaturesScanline ComputeSurfaceFeaturesScanline& operator=(const ComputeSurfaceFeaturesScanline&) = delete; ComputeSurfaceFeaturesScanline& operator=(ComputeSurfaceFeaturesScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized surface feature identification using a + * 3-slice rolling window with copyIntoBuffer bulk I/O. + * + * Handles both 3D and 2D geometries within a single Z-iteration loop, + * using coordinate remapping for 2D cases. + * + * @return Result<> indicating success, errors, or unsupported dimensionality. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all data. + const ComputeSurfaceFeaturesInputValues* m_InputValues = nullptr; ///< Algorithm parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cooperative cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message handler. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp index 38ec3da490..b92362f45e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.cpp @@ -52,7 +52,10 @@ class CropImageGeomDataArray auto srcDims = m_SrcImageGeom.getDimensions(); - // Copy one X-row at a time using bulk I/O + // OOC optimization: Copy one X-row at a time using bulk copyIntoBuffer/ + // copyFromBuffer. The original code used per-element getValue/setValue in + // a triple-nested loop, causing O(voxels * components) chunk operations. + // Row-at-a-time reduces this to O(Z * Y) bulk operations. const uint64 rowTuples = m_Bounds[1] - m_Bounds[0]; const usize rowElements = rowTuples * numComps; auto rowBuffer = std::make_unique(rowElements); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.hpp index 9366d04b8d..f58281d666 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/CropImageGeometry.hpp @@ -16,34 +16,49 @@ namespace nx::core { +/** + * @struct CropImageGeometryInputValues + * @brief Holds all user-configured parameters for the CropImageGeometry algorithm. + */ struct SIMPLNXCORE_EXPORT CropImageGeometryInputValues { - GeometrySelectionParameter::ValueType InputImageGeometryPath; - DataGroupCreationParameter::ValueType OutputImageGeometryPath; - ArraySelectionParameter::ValueType FeatureIdsPath; - VectorUInt64Parameter::ValueType MinVoxel; - VectorUInt64Parameter::ValueType MaxVoxel; - BoolParameter::ValueType RenumberFeatures; - AttributeMatrixSelectionParameter::ValueType CellFeatureAttributeMatrixPath; - BoolParameter::ValueType RemoveOriginalGeometry; - BoolParameter::ValueType CropXDim; - BoolParameter::ValueType CropYDim; - BoolParameter::ValueType CropZDim; + GeometrySelectionParameter::ValueType InputImageGeometryPath; ///< Source ImageGeom to crop. + DataGroupCreationParameter::ValueType OutputImageGeometryPath; ///< Destination path for the cropped geometry. + ArraySelectionParameter::ValueType FeatureIdsPath; ///< Per-cell Feature ID array (for renumbering). + VectorUInt64Parameter::ValueType MinVoxel; ///< User-specified minimum voxel bounds. + VectorUInt64Parameter::ValueType MaxVoxel; ///< User-specified maximum voxel bounds. + BoolParameter::ValueType RenumberFeatures; ///< If true, renumber Feature IDs to be contiguous. + AttributeMatrixSelectionParameter::ValueType CellFeatureAttributeMatrixPath; ///< Feature-level AM (for renumbering). + BoolParameter::ValueType RemoveOriginalGeometry; ///< If true, remove the source geometry after cropping. + BoolParameter::ValueType CropXDim; ///< Enable cropping in the X dimension. + BoolParameter::ValueType CropYDim; ///< Enable cropping in the Y dimension. + BoolParameter::ValueType CropZDim; ///< Enable cropping in the Z dimension. // Precomputed bounds from preflight - uint64 XMin; - uint64 XMax; - uint64 YMin; - uint64 YMax; - uint64 ZMin; - uint64 ZMax; + uint64 XMin; ///< Effective minimum X voxel index (inclusive). + uint64 XMax; ///< Effective maximum X voxel index (exclusive). + uint64 YMin; ///< Effective minimum Y voxel index (inclusive). + uint64 YMax; ///< Effective maximum Y voxel index (exclusive). + uint64 ZMin; ///< Effective minimum Z voxel index (inclusive). + uint64 ZMax; ///< Effective maximum Z voxel index (exclusive). }; /** * @class CropImageGeometry - * @brief This algorithm implements support code for the CropImageGeometryFilter + * @brief Crops a region of interest from an ImageGeom by copying voxel data + * from the source bounds into a new (smaller) ImageGeom. + * + * @section ooc_optimization Out-of-Core Optimization + * The original implementation copied data element-by-element using getValue()/setValue() + * in a triple-nested loop (Z, Y, X). For OOC data, each getValue() and setValue() + * call triggered a chunk operation. + * + * The optimized implementation copies data one X-row at a time using bulk + * copyIntoBuffer() and copyFromBuffer(). For each (Z, Y) pair, an entire row + * of (XMax - XMin) tuples is read/written in a single operation. This reduces + * the number of chunk operations from O(voxels * components) to O(Z * Y), a + * factor-of-XDim improvement. */ - class SIMPLNXCORE_EXPORT CropImageGeometry { public: @@ -55,13 +70,17 @@ class SIMPLNXCORE_EXPORT CropImageGeometry CropImageGeometry& operator=(const CropImageGeometry&) = delete; CropImageGeometry& operator=(CropImageGeometry&&) noexcept = delete; + /** + * @brief Executes the crop operation, copying data row-by-row via bulk I/O. + * @return Result<> indicating success or error. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const CropImageGeometryInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const CropImageGeometryInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp index 81ff9eac6a..ab8521f1c6 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.cpp @@ -8,6 +8,18 @@ using namespace nx::core; +// ============================================================================= +// DBSCAN — Dispatcher +// +// This file contains only the dispatch logic. The actual algorithm implementations +// live in DBSCANDirect.cpp (in-core) and DBSCANScanline.cpp (out-of-core). +// +// The dispatch checks both the ClusteringArray and FeatureIds array storage types: +// if either uses chunked on-disk storage (OOC), the Scanline variant is selected +// to avoid chunk thrashing during the multi-pass grid construction and distance +// computation phases. +// ============================================================================= + // ----------------------------------------------------------------------------- DBSCAN::DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues) : m_DataStructure(dataStructure) @@ -21,8 +33,19 @@ DBSCAN::DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesg DBSCAN::~DBSCAN() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to the appropriate algorithm variant based on storage type. + * + * Uses DispatchAlgorithm() to check whether the ClusteringArray + * or FeatureIds array is backed by out-of-core (chunked) storage. If so, the + * Scanline variant is used; otherwise, the Direct variant is selected. + * + * Both variants receive identical constructor arguments and produce identical output. + */ Result<> DBSCAN::operator()() { + // Check both arrays — the clustering array is read multiple times during grid + // construction (bounds, binning, filling), and featureIds is written during labeling. auto* clusteringArray = m_DataStructure.getDataAs(m_InputValues->ClusteringArrayPath); auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); return DispatchAlgorithm({clusteringArray, featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp index f2e6d9e070..dbc290e0cc 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCAN.hpp @@ -12,25 +12,64 @@ namespace nx::core { +/** + * @struct DBSCANInputValues + * @brief Input parameter bundle for the DBSCAN algorithm. + * + * Aggregates all DataPaths and configuration values needed by both the in-core + * (Direct) and out-of-core (Scanline) variants of DBSCAN clustering. + */ struct SIMPLNXCORE_EXPORT DBSCANInputValues { - DataPath ClusteringArrayPath; - DataPath MaskArrayPath; - DataPath FeatureIdsArrayPath; - float32 Epsilon; - int32 MinPoints; - ClusterUtilities::DistanceMetric DistanceMetric; - DataPath FeatureAM; - ChoicesParameter::ValueType ParseOrder; - std::mt19937_64::result_type Seed; + DataPath ClusteringArrayPath; ///< Input array containing 2D or 3D coordinate data to cluster + DataPath MaskArrayPath; ///< Input Bool/UInt8 mask; false elements become outliers (cluster 0) + DataPath FeatureIdsArrayPath; ///< Output Int32 array storing per-element cluster assignments + float32 Epsilon; ///< Maximum distance for density-connectivity; also determines grid cell size + int32 MinPoints; ///< Minimum points in a grid cell for it to be a "core" cell + ClusterUtilities::DistanceMetric DistanceMetric; ///< Distance metric used for canMerge checks between grid cells + DataPath FeatureAM; ///< Output Attribute Matrix resized to (maxCluster + 1) after clustering + ChoicesParameter::ValueType ParseOrder; ///< Order for processing core grids: LowDensityFirst, Random, or SeededRandom + std::mt19937_64::result_type Seed; ///< Random seed for reproducible parse order (SeededRandom mode) }; /** - * @class + * @class DBSCAN + * @brief Dispatcher algorithm for grid-based DBSCAN density clustering. + * + * Implements a modified DBSCAN algorithm based on Grid-based DBSCAN (GDCF) from + * Boonchoo et al. 2019. Data points are binned into a regular grid with cell side + * length epsilon / sqrt(dims). Grid cells with >= minPoints are "core" cells that + * form initial clusters. Adjacent grid cells are merged if any pair of points across + * them has distance < epsilon. + * + * This class acts as a thin dispatcher that selects between two concrete implementations: + * + * - **DBSCANDirect** (in-core): Uses per-element operator[] access for grid construction + * and direct random access for canMerge distance checks. Optimal when all arrays + * reside in memory. + * + * - **DBSCANScanline** (out-of-core / OOC): Uses chunked copyIntoBuffer() bulk I/O + * for grid construction (bounds detection, binning, cell filling). For canMerge + * distance checks, reads grid cell coordinate data on-demand into local buffers + * instead of random per-element access across the full array. + * + * The dispatch decision is made by DispatchAlgorithm() in + * AlgorithmDispatch.hpp, which checks whether any input IDataArray uses OOC storage. + * + * @see DBSCANDirect + * @see DBSCANScanline + * @see AlgorithmDispatch.hpp */ class SIMPLNXCORE_EXPORT DBSCAN { public: + /** + * @brief Constructs the dispatcher with all resources needed by either algorithm variant. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ DBSCAN(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, DBSCANInputValues* inputValues); ~DBSCAN() noexcept; @@ -39,20 +78,28 @@ class SIMPLNXCORE_EXPORT DBSCAN DBSCAN& operator=(const DBSCAN&) = delete; DBSCAN& operator=(DBSCAN&&) noexcept = delete; + /** + * @enum ParseOrder + * @brief Controls the order in which core grid cells are processed during initial clustering. + */ enum ParseOrder { - LowDensityFirst, - Random, - SeededRandom + LowDensityFirst, ///< Process lower-density core grids first (deterministic, typically fastest) + Random, ///< Process in non-deterministic random order (time-based seed) + SeededRandom ///< Process in deterministic random order (user-supplied seed) }; + /** + * @brief Dispatches to the Direct or Scanline algorithm based on storage type. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const DBSCANInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const DBSCANInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp index 69208d4d33..84b6e7062c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.cpp @@ -14,6 +14,40 @@ using namespace nx::core; +// ============================================================================= +// DBSCANDirect — In-Core Algorithm +// +// This file implements the in-core (Direct) variant of DBSCAN. +// It is selected by DispatchAlgorithm when all input arrays reside in memory. +// +// ALGORITHM OVERVIEW (Grid-based DBSCAN / GDCF): +// Based on "Grid-based DBSCAN: Indexing and inference" by Boonchoo et al. 2019. +// +// Phase 1 — Grid Construction: +// 1. Scan the input array to find min/max bounds per dimension +// 2. Create a regular grid with cell side length = epsilon / sqrt(dims) +// 3. Bin each data point into a grid cell +// 4. Build a compressed grid index (only non-empty cells) +// 5. Create per-axis bitmap tables for fast neighbor grid queries +// +// Phase 2 — Clustering: +// 1. Identify "core" grid cells (those with >= minPoints data points) +// 2. Sort core cells by parse order (low density first, random, etc.) +// 3. For each core cell, query neighbor grids and merge if canMerge +// returns true (any pair of points has distance < epsilon) +// 4. Expand clusters to border (non-core) grid cells +// +// Phase 3 — Labeling: +// Write the final cluster ID for each data point based on its grid cell's +// cluster assignment. Points in unassigned cells become outliers (cluster 0). +// +// DATA ACCESS PATTERN: +// Uses direct operator[] for per-element random access. Grid construction +// requires 2-3 full passes over the input array. canMerge requires random +// access to arbitrary tuple indices within grid cells. Both patterns are +// optimal for in-memory data but would cause chunk thrashing for OOC data. +// ============================================================================= + namespace { /** @@ -879,7 +913,11 @@ class GDCF QuickSortGrids(sorted, next + 1, end); } - // In-core path: direct random access via operator[] is fast + // In-core path: direct random access via operator[] is fast. + // For each pair of points (one from each grid cell), compute the distance + // and return true as soon as any pair is within epsilon. With in-memory data, + // operator[] is a pointer dereference, so random access to arbitrary tuple + // indices is efficient. For OOC data, see DBSCANScanline's readGridCellCoords(). bool canMerge(usize pGridId, usize qGridId) { for(usize pPointId : hyperGridBitMap.gridVoxels[pGridId]) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp index 2f8edee2f0..a107aa1ac4 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANDirect.hpp @@ -11,13 +11,33 @@ struct DBSCANInputValues; /** * @class DBSCANDirect - * @brief In-core algorithm for DBSCAN. Uses direct per-element getValue()/operator[] - * access for grid construction and canMerge distance computation. Selected by - * DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * @brief In-core algorithm for grid-based DBSCAN using direct per-element array access. + * + * Uses operator[] for all data access phases: + * - Grid construction: reads each tuple's coordinates to compute bounds and bin assignments + * - Distance computation (canMerge): directly indexes into the input array by tuple index + * for pairwise distance checks between grid cell members + * - Labeling: writes cluster IDs via setValue() + * + * This is optimal when all arrays reside in memory, where operator[] is essentially a + * pointer dereference. For out-of-core data, each operator[] call may trigger chunk + * load/evict cycles, so DBSCANScanline should be used instead. + * + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * + * @see DBSCANScanline for the out-of-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT DBSCANDirect { public: + /** + * @brief Constructs the in-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ DBSCANDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const DBSCANInputValues* inputValues); ~DBSCANDirect() noexcept; @@ -26,13 +46,17 @@ class SIMPLNXCORE_EXPORT DBSCANDirect DBSCANDirect& operator=(const DBSCANDirect&) = delete; DBSCANDirect& operator=(DBSCANDirect&&) noexcept = delete; + /** + * @brief Executes the in-core DBSCAN clustering: grid construction, clustering, labeling. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const DBSCANInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const DBSCANInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp index eacf822bba..3e17d95228 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.cpp @@ -15,6 +15,43 @@ using namespace nx::core; +// ============================================================================= +// DBSCANScanline — Out-of-Core (OOC) Algorithm +// +// This file implements the out-of-core (Scanline) variant of DBSCAN. +// It is selected by DispatchAlgorithm when any input array uses chunked on-disk +// storage (e.g., ZarrStore / HDF5 chunked store). +// +// PROBLEM: +// The DBSCAN algorithm requires multiple passes over the input array during +// grid construction (bounds, binning, filling) and random access to arbitrary +// tuple indices during canMerge distance checks. When data is stored out-of-core, +// each operator[] call may trigger a chunk load/decompress/evict cycle. The grid +// construction passes iterate over all N tuples 2-3 times, and canMerge checks +// access random positions in the array, making both phases vulnerable to chunk +// thrashing. +// +// SOLUTION: +// 1. Grid construction uses 64K-tuple chunk reads via copyIntoBuffer(). Each +// of the 2-3 passes reads the input array sequentially in chunks, processing +// all tuples in each chunk before moving to the next. This converts N per- +// element random accesses into N/65536 sequential bulk reads. +// +// 2. canMerge uses readGridCellCoords() to bulk-read all coordinate data for +// each grid cell into a local float32 buffer. The pairwise distance check +// then operates entirely on in-memory data. Memory cost per canMerge call +// is O(gridCellSize * dims), which is typically small (grid cells contain +// just a handful of points in practice). +// +// 3. The clustering and labeling phases operate on the in-memory grid index +// (gridVoxels, clusterForest), which is identical to the Direct variant. +// Only grid construction and canMerge are modified for OOC. +// +// NOTE: The HyperGridBitMap constructors in this file do NOT accept a +// MessageHelper parameter (unlike the Direct variant) because they were +// written to minimize the parameter surface for the OOC path. +// ============================================================================= + namespace { /** @@ -860,8 +897,20 @@ class GDCF const std::atomic_bool& m_ShouldCancel; /** - * @brief Reads coordinate data for all points in a grid cell from the store. - * Memory cost is O(gridCellSize * dims), not O(n). + * @brief Reads coordinate data for all points in a grid cell from the OOC store. + * + * Instead of random operator[] access to arbitrary tuple indices scattered across + * the full input array (which would cause chunk thrashing), this method reads each + * grid cell member's coordinates via single-tuple copyIntoBuffer() calls and + * assembles them into a contiguous local buffer. + * + * Memory cost is O(gridCellSize * dims) per call, which is typically very small + * (grid cells contain a handful of points in practice). The returned buffer is + * used for all pairwise distance computations in canMerge, so the data is read + * once and reused for every comparison. + * + * @param gridId Index of the grid cell whose member coordinates to read + * @return Contiguous float32 buffer with coordinates for all grid cell members */ std::vector readGridCellCoords(usize gridId) const { @@ -924,7 +973,11 @@ class GDCF QuickSortGrids(sorted, next + 1, end); } - // OOC path: read grid cell coords on-demand into O(gridCellSize) local buffers + // OOC path: read grid cell coords on-demand into O(gridCellSize) local buffers. + // Both grid cells' coordinate data is read in full via readGridCellCoords(), + // then all pairwise distances are computed entirely in memory. This avoids + // random operator[] access to the full OOC input array, which would trigger + // chunk load/evict cycles for every single distance computation. bool canMerge(usize pGridId, usize qGridId) { const usize dims = static_cast(HGBPT::Dimensions); @@ -1038,6 +1091,9 @@ Result<> DBSCANScanline::operator()() return {}; } + // OOC: find max cluster ID using chunked bulk I/O instead of std::max_element + // on the OOC store (which would use per-element iterator access). Read featureIds + // in 1M-element chunks and track the maximum across all chunks. auto& featureIdsDataStore = featureIds.getDataStoreRef(); int32 maxCluster = 0; { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp index b4da78384d..5ae4703553 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/DBSCANScanline.hpp @@ -11,14 +11,41 @@ struct DBSCANInputValues; /** * @class DBSCANScanline - * @brief Out-of-core algorithm for DBSCAN. Uses chunked copyIntoBuffer bulk I/O - * for grid construction and on-demand per-grid-cell reads for canMerge distance - * computation. Selected by DispatchAlgorithm when any input array is backed by - * ZarrStore (out-of-core storage). + * @brief Out-of-core algorithm for grid-based DBSCAN using chunked bulk I/O. + * + * The DBSCAN algorithm has two major data access phases that benefit from OOC optimization: + * + * **Grid construction** (bounds detection, binning, cell filling): The input array is + * scanned multiple times to compute min/max bounds, bin each point into a grid cell, + * and build the grid-to-point index. The Scanline variant reads the input array in + * sequential 64K-tuple chunks via copyIntoBuffer(), amortizing OOC overhead across + * thousands of elements per read instead of per-element random access. + * + * **Distance computation (canMerge)**: When checking if two adjacent grid cells should + * merge, pairwise distances between all points in both cells are computed. The Direct + * variant uses operator[] to index directly into the full input array by tuple index. + * The Scanline variant instead uses readGridCellCoords() to bulk-read all coordinate + * data for each grid cell into a local buffer, then performs pairwise distances entirely + * in memory. Memory cost is O(gridCellSize * dims) per canMerge call, not O(n). + * + * **Labeling**: Both variants use setValue() for writing cluster IDs, which is acceptable + * since labeling is a single sequential pass. + * + * Selected by DispatchAlgorithm when any input array is backed by out-of-core storage. + * + * @see DBSCANDirect for the in-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT DBSCANScanline { public: + /** + * @brief Constructs the out-of-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ DBSCANScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const DBSCANInputValues* inputValues); ~DBSCANScanline() noexcept; @@ -27,13 +54,17 @@ class SIMPLNXCORE_EXPORT DBSCANScanline DBSCANScanline& operator=(const DBSCANScanline&) = delete; DBSCANScanline& operator=(DBSCANScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized DBSCAN clustering: grid construction, clustering, labeling. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const DBSCANInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const DBSCANInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp index c3a0014387..6e7983b664 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.cpp @@ -1,3 +1,39 @@ +/** + * @file ErodeDilateBadData.cpp + * @brief Iterative morphological erosion/dilation of bad (FeatureId == 0) voxels, + * optimized for out-of-core (OOC) data stores via Z-slice buffered I/O. + * + * ## High-Level Flow (per iteration) + * + * 1. **Initialize rolling window** -- Load FeatureId Z-slices 0 and 1 into + * slots 1 (current) and 2 (next) of the three-element window. + * + * 2. **Scan every voxel** (Z-major, then Y, then X): + * - For each bad voxel (featureId == 0), examine its 6 face neighbors using + * the in-memory rolling-window buffers. + * - **Dilate**: If a neighbor is good (featureId > 0), mark that good + * neighbor to be overwritten with the bad voxel's data. + * - **Erode**: Tally the good neighbors and record the most common one; + * mark the bad voxel to be overwritten with that neighbor's data. + * - Marks are stored in three O(sliceSize) arrays (one per rolling-window + * slot) rather than a single O(totalPoints) array. + * + * 3. **Deferred transfer** -- After processing each Z-slice, the marks for + * slice z-1 are complete (because only voxels at z-2, z-1, and z can + * affect z-1). Commit those marks via SliceBufferedTransferOneZ, which + * reads the source and destination slices in bulk, applies the copies + * in-memory, and writes back. + * + * 4. **Rotate windows** -- Swap mark arrays and FeatureId slices forward by + * one position. Clear the newly vacated slot for the next Z-layer. + * + * 5. **Flush final slice** -- After the Z-loop exits, the last slice's marks + * are still pending; commit them. + * + * Each iteration must re-read FeatureIds from the store because the preceding + * iteration's SliceBufferedTransferOneZ calls may have changed values. + */ + #include "ErodeDilateBadData.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -36,12 +72,15 @@ Result<> ErodeDilateBadData::operator()() SizeVec3 udims = selectedImageGeom.getDimensions(); std::array dims = {static_cast(udims[0]), static_cast(udims[1]), static_cast(udims[2])}; + // Precompute face-neighbor offsets: given a flat voxel index, adding one of + // these offsets yields the flat index of the corresponding face neighbor. std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); - // Find max feature ID using Z-slice batched reads + // ---- Determine max FeatureId using sequential Z-slice reads ---- + // This avoids a full-volume random-access scan that would thrash OOC chunks. usize numFeatures = 0; { std::vector sliceBuf(sliceSize); @@ -58,9 +97,15 @@ Result<> ErodeDilateBadData::operator()() } } + // featureCount is used during erosion to tally how many of each neighbor + // FeatureId surround a bad voxel. It is sized to numFeatures+1 so that + // featureCount[featureId] is directly addressable. std::vector featureCount(numFeatures + 1, 0); - // FeatureIds rolling window for neighbor lookups + // ---- FeatureIds rolling window (3 Z-slices) ---- + // Slot 0 = z-1, slot 1 = z (current), slot 2 = z+1. + // All face-neighbor FeatureId reads come from these buffers rather than + // the underlying (potentially OOC) data store. std::array, 3> featureIdSlices; for(auto& fis : featureIdSlices) { @@ -69,21 +114,31 @@ Result<> ErodeDilateBadData::operator()() auto readFeatureIdSlice = [&](int64 z, usize slot) { featureIds.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(featureIdSlices[slot].data(), sliceSize)); }; + // Maps face-neighbor index (0=-Z, 1=-Y, 2=-X, 3=+X, 4=+Y, 5=+Z) to the + // rolling-window slot that holds the neighbor's Z-slice. + // -Z is in slot 0 (prev), -Y/-X/+X/+Y are in slot 1 (current), +Z is in slot 2 (next). constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; - // Per-slice mark arrays: marks[0]=z-1, marks[1]=z, marks[2]=z+1 - // Each entry is -1 (no transfer) or the global source index. - // This replaces the O(totalPoints) neighbors array with O(3*sliceSize). + // ---- Per-slice mark arrays (3 x sliceSize) ---- + // Each entry is -1 (no transfer needed) or the global flat index of the + // source voxel whose data should be copied to this position. + // marks[0] corresponds to Z-1, marks[1] to Z, marks[2] to Z+1. + // This replaces a full-volume O(totalPoints) neighbor array with + // O(3 * sliceSize) memory, which is critical for large datasets. std::array, 3> marks; for(auto& m : marks) { m.resize(sliceSize); } + // Collect all sibling arrays in the same Attribute Matrix as FeatureIds, + // excluding user-specified ignored arrays. These will all be updated + // during the transfer phase to stay consistent with FeatureId changes. const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsArrayPath, m_InputValues->IgnoredDataArrayPaths); const usize dimZ = static_cast(dims[2]); - // Helper to transfer a single Z-slice across all arrays + // Commits one Z-slice of marks across all sibling arrays. + // SliceBufferedTransferOneZ handles the bulk read/copy/write internally. auto transferSlice = [&](usize z, const std::vector& sliceMarks) { for(const auto& voxelArray : voxelArrays) { @@ -91,24 +146,31 @@ Result<> ErodeDilateBadData::operator()() } }; + // ---- Main iteration loop ---- + // Each iteration performs one complete pass of the morphological operation. + // The rolling window and mark arrays are reset at the start of each + // iteration because the previous pass's transfers alter the data. for(int32 iteration = 0; iteration < m_InputValues->NumIterations; iteration++) { - // Clear marks + // Clear all mark arrays for this iteration for(auto& m : marks) { std::fill(m.begin(), m.end(), -1); } - // Initialize FeatureId rolling window + // Initialize FeatureId rolling window: z=0 -> slot 1, z=1 -> slot 2 readFeatureIdSlice(0, 1); if(dims[2] > 1) { readFeatureIdSlice(1, 2); } + // ---- Z-slice scan loop ---- for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - // Advance FeatureId rolling window + // Advance the rolling window forward by one Z-slice. + // After this swap: slot 0 = old current (z-1), slot 1 = old next (z), + // slot 2 = newly loaded z+1. if(zIdx > 0) { std::swap(featureIdSlices[0], featureIdSlices[1]); @@ -119,16 +181,21 @@ Result<> ErodeDilateBadData::operator()() } } - // Find neighbors for bad voxels in this Z-slice + // ---- Inner XY scan: identify marks for bad voxels in this Z-slice ---- for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { const usize inSlice = static_cast(yIdx * dims[0] + xIdx); const int32 featureName = featureIdSlices[1][inSlice]; + + // Only process bad voxels (featureId == 0) if(featureName == 0) { int32 most = 0; + + // Determine which face neighbors are within the volume bounds, + // then mask off directions the user has disabled. std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); if(!m_InputValues->XDirOn) { @@ -146,7 +213,13 @@ Result<> ErodeDilateBadData::operator()() isValidFaceNeighbor[k_PositiveZNeighbor] = false; } + // Global flat index of this voxel (used as source index in marks) const int64 voxelIndex = xIdx + yIdx * dims[0] + zIdx * static_cast(sliceSize); + + // Map each face neighbor to its position within the appropriate + // rolling-window slice buffer. For -Z and +Z the XY position is + // the same (inSlice); for in-plane neighbors it shifts by +/-1 + // in the X or Y direction. const std::array neighborInSlice = { inSlice, // -Z static_cast((yIdx - 1) * dims[0] + xIdx), // -Y @@ -162,27 +235,35 @@ Result<> ErodeDilateBadData::operator()() { continue; } + // Compute the global flat index of this neighbor and read its + // FeatureId from the rolling-window buffer (not the OOC store). const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; const int32 feature = featureIdSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; if(m_InputValues->Operation == detail::k_DilateIndex && feature > 0) { - // Mark the good NEIGHBOR to be overwritten by this bad voxel. - // The neighbor is in slot k_NeighborSlot[faceIndex] (0=z-1, 1=z, 2=z+1). + // DILATION: Mark the good NEIGHBOR to receive this bad voxel's + // data. The neighbor lives in rolling-window slot + // k_NeighborSlot[faceIndex], so we mark that slot's array. marks[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] = voxelIndex; } if(feature > 0 && m_InputValues->Operation == detail::k_ErodeIndex) { + // EROSION: Tally this good neighbor's FeatureId and track + // which one is the most common ("majority vote"). featureCount[feature]++; const int32 current = featureCount[feature]; if(current > most) { most = current; - // Mark this bad voxel to be overwritten by the best neighbor + // Mark this bad voxel to receive the best neighbor's data marks[1][inSlice] = neighborPoint; } } } + + // Reset featureCount entries for this voxel's neighbors so the + // array is clean for the next bad voxel (avoids a full memset). if(m_InputValues->Operation == detail::k_ErodeIndex) { for(const auto& faceIndex : faceNeighborInternalIdx) @@ -199,19 +280,27 @@ Result<> ErodeDilateBadData::operator()() } } - // Transfer z-1: all marks for z-1 are now complete (from bad voxels at z-2, z-1, z) + // ---- Deferred transfer for slice z-1 ---- + // At this point all voxels that could write marks into slice z-1 have + // been processed (only voxels in slices z-2, z-1, and z can affect z-1 + // via face-neighbor relationships). It is now safe to commit z-1. if(zIdx > 0) { transferSlice(static_cast(zIdx - 1), marks[0]); } - // Rotate marks: [0]=old[1], [1]=old[2], [2]=cleared + // ---- Rotate mark arrays forward ---- + // marks[0] <- old marks[1] (becomes the "previous" for next iteration) + // marks[1] <- old marks[2] (becomes the "current") + // marks[2] <- cleared (ready for the next Z+1 layer) std::swap(marks[0], marks[1]); std::swap(marks[1], marks[2]); std::fill(marks[2].begin(), marks[2].end(), -1); } - // Transfer last slice + // ---- Flush final Z-slice ---- + // After the Z-loop exits, the last slice's marks are in marks[0] (due to + // the rotation) and have not yet been committed. if(dims[2] > 0) { transferSlice(static_cast(dims[2] - 1), marks[0]); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp index 10ccc8d537..2ceb83ad2d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateBadData.hpp @@ -23,22 +23,73 @@ static inline constexpr ChoicesParameter::ValueType k_ErodeIndex = 1ULL; /** * @struct ErodeDilateBadDataInputValues * @brief Holds all user-supplied parameters for the ErodeDilateBadData algorithm. + * + * This struct is populated by the filter's preflight/execute methods and passed + * into the algorithm so that the algorithm itself has no dependency on the + * parameter system. */ struct SIMPLNXCORE_EXPORT ErodeDilateBadDataInputValues { - ChoicesParameter::ValueType Operation; - int32 NumIterations; - bool XDirOn; - bool YDirOn; - bool ZDirOn; - DataPath FeatureIdsArrayPath; - MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; - DataPath InputImageGeometry; + ChoicesParameter::ValueType Operation; ///< Morphological operation: detail::k_DilateIndex (0) or detail::k_ErodeIndex (1) + int32 NumIterations; ///< Number of erosion/dilation passes to perform + bool XDirOn; ///< Whether to consider neighbors along the X axis + bool YDirOn; ///< Whether to consider neighbors along the Y axis + bool ZDirOn; ///< Whether to consider neighbors along the Z axis + DataPath FeatureIdsArrayPath; ///< Path to the Int32 FeatureIds cell array (0 = bad data) + MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; ///< Arrays excluded from the data transfer phase + DataPath InputImageGeometry; ///< Path to the ImageGeom that defines the voxel grid }; /** * @class ErodeDilateBadData - * @brief Erodes or dilates bad (FeatureId == 0) voxels by replacing them with the most common neighbor feature. + * @brief Iterative morphological erosion or dilation of "bad" voxels (FeatureId == 0) + * on a structured ImageGeom grid, optimized for out-of-core (OOC) data stores. + * + * ## Algorithm Overview + * + * **Bad data** is any cell whose FeatureId is 0, meaning it failed some prior + * classification step. This algorithm either grows those bad regions (dilation) + * or shrinks them (erosion) by one voxel per iteration, repeating for a + * user-specified number of iterations. + * + * - **Dilation**: For every bad voxel that has a non-zero (good) face neighbor, + * the good neighbor is marked to become bad. This expands the bad region + * outward by one cell. + * - **Erosion**: For every bad voxel, the surrounding good neighbors are tallied + * and the most common FeatureId is chosen. The bad voxel is replaced with + * that FeatureId, shrinking the bad region inward by one cell. + * + * After marks are determined, all sibling data arrays in the same Attribute + * Matrix (except those in the ignored list) are updated in bulk using + * SliceBufferedTransferOneZ, so that every array stays consistent with the + * modified FeatureIds. + * + * ## OOC Optimization Strategy + * + * When data resides in an out-of-core (chunked, disk-backed) store, random + * voxel access triggers expensive chunk load/evict cycles ("chunk thrashing"). + * This implementation avoids that by: + * + * 1. **3-slice rolling window for FeatureIds**: Three Z-slices (prev, current, + * next) are held in contiguous std::vector buffers, loaded via + * copyIntoBuffer(). Face-neighbor lookups index into these buffers instead + * of hitting the OOC store. As the Z loop advances, the window shifts + * forward with O(1) pointer swaps. + * + * 2. **Per-slice mark arrays instead of a full-volume neighbor array**: Classic + * implementations allocate an O(totalPoints) neighbors array. This version + * uses three O(sliceSize) mark arrays (one per Z-slice in the window), + * reducing peak memory from O(volume) to O(3 * sliceSize). + * + * 3. **Deferred, sequential Z-slice writes**: Marks are accumulated as the + * scan proceeds. When slice z finishes, the marks for slice z-1 are + * guaranteed complete (no future voxel can modify them), so z-1 is + * transferred via SliceBufferedTransferOneZ. This keeps writes sequential + * and aligned with OOC chunk boundaries. + * + * 4. **Per-iteration re-read**: Each iteration re-initializes the rolling + * window from the store because the previous iteration's transfers may + * have changed FeatureId values. */ class SIMPLNXCORE_EXPORT ErodeDilateBadData { @@ -64,17 +115,28 @@ class SIMPLNXCORE_EXPORT ErodeDilateBadData /** * @brief Executes the erode/dilate bad data algorithm. + * + * Runs NumIterations passes of the selected morphological operation + * (erosion or dilation) over the entire ImageGeom volume. Each pass + * processes all Z-slices sequentially using a 3-slice rolling window + * for FeatureId lookups and deferred SliceBufferedTransferOneZ calls + * for the data-copy phase. + * * @return Result<> indicating success or any errors encountered during execution */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag. + * @return const reference to the atomic cancellation flag + */ const std::atomic_bool& getCancel() const; private: - DataStructure& m_DataStructure; - const ErodeDilateBadDataInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure holding all arrays + const ErodeDilateBadDataInputValues* m_InputValues = nullptr; ///< User-supplied algorithm parameters + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked between iterations + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress/status messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp index 7f48fd1b8b..7fa39b5a14 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.cpp @@ -1,3 +1,33 @@ +/** + * @file ErodeDilateCoordinationNumber.cpp + * @brief Coordination-number-based boundary smoothing of good/bad voxels, + * optimized for out-of-core (OOC) data stores via Z-slice buffered I/O. + * + * ## High-Level Flow (per pass) + * + * 1. **Initialize rolling window** -- Load FeatureId Z-slices 0 and 1 into + * the three-element window (slots 1 and 2). + * + * 2. **Scan every voxel** (Z-major, then Y, then X): + * - For each voxel on a good/bad boundary, count face neighbors of the + * opposite type (the "coordination number") and record the most common + * neighbor FeatureId. + * - Store the coordination number and best-neighbor index in per-slice + * arrays rather than full-volume arrays. + * + * 3. **Deferred transfer** -- After processing Z-slice z, commit the marks + * for z-1 (which are now complete). Only voxels whose coordination number + * meets or exceeds the user's threshold are actually transferred. + * + * 4. **Rotate windows** -- Shift rolling-window buffers and per-slice arrays + * forward by one Z-layer. + * + * 5. **Flush final slice** -- Commit the last slice's marks after the Z-loop. + * + * 6. **Repeat** until no voxels were modified (if Loop is true) or after + * one pass (if Loop is false). + */ + #include "ErodeDilateCoordinationNumber.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -37,15 +67,18 @@ Result<> ErodeDilateCoordinationNumber::operator()() SizeVec3 udims = selectedImageGeom.getDimensions(); std::array dims = {static_cast(udims[0]), static_cast(udims[1]), static_cast(udims[2])}; + // Precompute face-neighbor index offsets and iteration order std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); + // Collect all sibling arrays that should be updated during the transfer phase const std::vector> voxelArrays = nx::core::GenerateDataArrayList(m_DataStructure, m_InputValues->FeatureIdsArrayPath, m_InputValues->IgnoredDataArrayPaths); const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); const usize dimZ = static_cast(dims[2]); - // Find max feature ID using Z-slice batched reads + // ---- Determine max FeatureId using sequential Z-slice reads ---- + // Sequential bulk reads avoid OOC chunk thrashing. const auto& featureIdsStore = featureIds.getDataStoreRef(); usize numFeatures = 0; { @@ -63,11 +96,14 @@ Result<> ErodeDilateCoordinationNumber::operator()() } } + // Per-voxel neighbor feature tally, sized so featureCount[featureId] is + // directly addressable. Reset after each voxel to avoid a full memset. std::vector featureCount(numFeatures + 1, 0); bool keepGoing = true; int32 counter = 1; - // FeatureIds rolling window + // ---- FeatureIds rolling window (3 Z-slices) ---- + // Slot 0 = z-1 (previous), slot 1 = z (current), slot 2 = z+1 (next). std::array, 3> featureIdSlices; for(auto& fis : featureIdSlices) { @@ -76,26 +112,34 @@ Result<> ErodeDilateCoordinationNumber::operator()() auto readFeatureIdSlice = [&](int64 z, usize slot) { featureIdsStore.copyIntoBuffer(static_cast(z) * sliceSize, nonstd::span(featureIdSlices[slot].data(), sliceSize)); }; + // Maps face-neighbor index to rolling-window slot: + // -Z -> slot 0, -Y/-X/+X/+Y -> slot 1 (same Z), +Z -> slot 2 constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; - // Per-slice neighbors (O(3*sliceSize) replaces O(totalPoints) neighbors array) - // Slot 0=z-1, slot 1=z, slot 2=z+1 + // ---- Per-slice neighbor marks: 3 x O(sliceSize) ---- + // Each entry is -1 (no transfer) or the global flat index of the source voxel. + // Replaces the O(totalPoints) full-volume neighbors array. std::array, 3> sliceNeighbors; for(auto& sn : sliceNeighbors) { sn.resize(sliceSize, -1); } - // Per-slice coordination numbers (O(3*sliceSize) replaces O(totalPoints)) + // ---- Per-slice coordination numbers: 3 x O(sliceSize) ---- + // Tracks the coordination number for each voxel in the rolling window. + // Only voxels whose coordination number meets the threshold will be + // transferred during the commit phase. std::array, 3> sliceCoordination; for(auto& sc : sliceCoordination) { sc.resize(sliceSize, 0); } - // Helper to transfer a single Z-slice across all arrays, for qualifying voxels + // Commits one Z-slice worth of marks, but only for voxels whose coordination + // number meets or exceeds the user's threshold. This filtering step is what + // distinguishes this algorithm from simple erosion/dilation: low-coordination + // boundary voxels are left alone. auto transferSlice = [&](usize z, const std::vector& marks, const std::vector& coord) { - // Filter marks: only transfer voxels meeting coordination threshold std::vector filteredMarks(sliceSize, -1); for(usize i = 0; i < sliceSize; i++) { @@ -111,6 +155,9 @@ Result<> ErodeDilateCoordinationNumber::operator()() } }; + // ---- Main pass loop ---- + // Repeats until either (a) no voxels were modified this pass, or + // (b) Loop is false and a single pass has completed. while(counter > 0 && keepGoing) { counter = 0; @@ -119,7 +166,7 @@ Result<> ErodeDilateCoordinationNumber::operator()() keepGoing = false; } - // Clear per-slice arrays + // Clear per-slice tracking arrays for this pass for(auto& sn : sliceNeighbors) { std::fill(sn.begin(), sn.end(), -1); @@ -129,15 +176,17 @@ Result<> ErodeDilateCoordinationNumber::operator()() std::fill(sc.begin(), sc.end(), 0); } - // Initialize rolling window + // Re-initialize rolling window from the (potentially modified) store readFeatureIdSlice(0, 1); if(dims[2] > 1) { readFeatureIdSlice(1, 2); } + // ---- Z-slice scan loop ---- for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { + // Advance the FeatureId rolling window if(zIdx > 0) { std::swap(featureIdSlices[0], featureIdSlices[1]); @@ -148,6 +197,7 @@ Result<> ErodeDilateCoordinationNumber::operator()() } } + // ---- Inner XY scan ---- for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) @@ -160,6 +210,7 @@ Result<> ErodeDilateCoordinationNumber::operator()() std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + // Map each face neighbor to its position within its rolling-window slice const std::array neighborInSlice = { inSlice, // -Z static_cast((yIdx - 1) * dims[0] + xIdx), // -Y @@ -179,6 +230,8 @@ Result<> ErodeDilateCoordinationNumber::operator()() const int64 neighborPoint = voxelIndex + neighborVoxelIndexOffsets[faceIndex]; const int32 feature = featureIdSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]]; + // A voxel is on the boundary if it and its neighbor have opposite + // good/bad status (one is 0, the other is > 0). if((featureName > 0 && feature == 0) || (featureName == 0 && feature > 0)) { coordination = coordination + 1; @@ -187,13 +240,15 @@ Result<> ErodeDilateCoordinationNumber::operator()() if(current > most) { most = current; + // Record this neighbor as the best replacement source sliceNeighbors[1][inSlice] = neighborPoint; } } } + // Store the computed coordination number for the transfer-filter step sliceCoordination[1][inSlice] = coordination; - // Reset featureCount for neighbors + // Reset featureCount entries touched by this voxel's neighbors for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) @@ -209,13 +264,14 @@ Result<> ErodeDilateCoordinationNumber::operator()() } } - // Transfer z-1 (complete after processing z) + // ---- Deferred transfer for slice z-1 ---- + // After processing slice z, all marks for z-1 are complete. if(zIdx > 0) { transferSlice(static_cast(zIdx - 1), sliceNeighbors[0], sliceCoordination[0]); } - // Rotate per-slice arrays + // ---- Rotate per-slice arrays forward ---- std::swap(sliceNeighbors[0], sliceNeighbors[1]); std::swap(sliceNeighbors[1], sliceNeighbors[2]); std::fill(sliceNeighbors[2].begin(), sliceNeighbors[2].end(), -1); @@ -225,7 +281,7 @@ Result<> ErodeDilateCoordinationNumber::operator()() std::fill(sliceCoordination[2].begin(), sliceCoordination[2].end(), 0); } - // Transfer last slice + // ---- Flush final Z-slice ---- if(dims[2] > 0) { transferSlice(static_cast(dims[2] - 1), sliceNeighbors[0], sliceCoordination[0]); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp index 214a165693..2092d8b433 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateCoordinationNumber.hpp @@ -16,19 +16,63 @@ namespace nx::core /** * @struct ErodeDilateCoordinationNumberInputValues * @brief Holds all user-supplied parameters for the ErodeDilateCoordinationNumber algorithm. + * + * Populated by the filter's preflight/execute methods and passed into the + * algorithm to decouple it from the parameter system. */ struct SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumberInputValues { - int32 CoordinationNumber; - bool Loop; - DataPath FeatureIdsArrayPath; - MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; - DataPath InputImageGeometry; + int32 CoordinationNumber; ///< Maximum tolerated coordination number. Voxels at a good/bad boundary + ///< whose coordination number meets or exceeds this value will be modified. + bool Loop; ///< If true, repeat the operation until no more voxels exceed the threshold + DataPath FeatureIdsArrayPath; ///< Path to the Int32 FeatureIds cell array (0 = bad data) + MultiArraySelectionParameter::ValueType IgnoredDataArrayPaths; ///< Arrays excluded from the data transfer phase + DataPath InputImageGeometry; ///< Path to the ImageGeom that defines the voxel grid }; /** * @class ErodeDilateCoordinationNumber - * @brief Smooths voxel boundaries by eroding or dilating based on coordination number thresholds. + * @brief Smooths voxel boundaries by eroding or dilating based on coordination + * number thresholds, optimized for out-of-core (OOC) data stores. + * + * ## Algorithm Overview + * + * The "coordination number" of a voxel on a good/bad boundary is the count of + * its 6 face neighbors that belong to the opposite class (good vs. bad, where + * bad means FeatureId == 0). A high coordination number indicates that a voxel + * is surrounded mostly by the opposite type and is therefore likely noise or a + * rough boundary artifact. + * + * For each voxel on the boundary: + * 1. Count the face neighbors of the opposite type (coordination number). + * 2. Among those opposite-type neighbors, find the most common FeatureId. + * 3. If the coordination number meets or exceeds the user's threshold, mark + * the voxel to be replaced by the most common neighbor's data. + * + * This is repeated until no voxels exceed the threshold (if Loop is true) or + * for a single pass (if Loop is false). + * + * ## OOC Optimization Strategy + * + * The same 3-slice rolling window and per-slice mark/coordination array pattern + * used in ErodeDilateBadData applies here: + * + * 1. **3-slice rolling window for FeatureIds**: Three Z-slices (prev, current, + * next) are held in std::vector buffers loaded via copyIntoBuffer(). All + * face-neighbor FeatureId lookups read from these buffers. + * + * 2. **Per-slice mark and coordination arrays**: Instead of full-volume arrays, + * three O(sliceSize) neighbor-mark arrays and three O(sliceSize) coordination + * arrays track the results for the rolling window. This reduces peak memory + * from O(volume) to O(sliceSize). + * + * 3. **Deferred sequential writes**: Marks for slice z-1 are committed after + * processing slice z, because only voxels at z-2 through z can affect z-1. + * The transfer is conditional: only voxels whose coordination number meets + * the threshold are actually committed, using SliceBufferedTransferOneZ. + * + * 4. **Per-pass re-read**: Each pass re-reads FeatureIds from the store because + * the previous pass's transfers may have changed boundary conditions. */ class SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumber { @@ -53,18 +97,29 @@ class SIMPLNXCORE_EXPORT ErodeDilateCoordinationNumber ErodeDilateCoordinationNumber& operator=(ErodeDilateCoordinationNumber&&) noexcept = delete; /** - * @brief Executes the erode/dilate coordination number algorithm. + * @brief Executes the coordination-number-based erosion/dilation algorithm. + * + * Runs one or more passes (depending on the Loop flag) over the entire + * ImageGeom volume. Each pass processes all Z-slices sequentially using a + * 3-slice rolling window, accumulating per-voxel coordination numbers and + * best-neighbor marks. After each Z-slice completes, the previous slice's + * qualifying marks are committed via SliceBufferedTransferOneZ. + * * @return Result<> indicating success or any errors encountered during execution */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag. + * @return const reference to the atomic cancellation flag + */ const std::atomic_bool& getCancel() const; private: - DataStructure& m_DataStructure; - const ErodeDilateCoordinationNumberInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure holding all arrays + const ErodeDilateCoordinationNumberInputValues* m_InputValues = nullptr; ///< User-supplied algorithm parameters + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked between passes + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress/status messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp index 32ed5f7cb8..4af69efd70 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.cpp @@ -1,3 +1,42 @@ +/** + * @file ErodeDilateMask.cpp + * @brief Iterative morphological erosion/dilation of a boolean mask array, + * optimized for out-of-core (OOC) data stores via Z-slice buffered I/O. + * + * ## High-Level Flow (per iteration) + * + * 1. **Initialize dual rolling windows** -- Load mask Z-slices 0 and 1 into + * both `maskSlices` (read buffer) and `maskCopySlices` (write buffer). + * + * 2. **Scan every voxel** (Z-major, then Y, then X): + * - For each false (unmasked) voxel, examine face neighbors using the + * read buffer (`maskSlices`). + * - **Dilate**: If any neighbor is true, set this voxel to true in the + * write buffer (`maskCopySlices[1]`). + * - **Erode**: If any neighbor is true, set that neighbor to false in + * the write buffer (`maskCopySlices[slot]`). + * - The dual-buffer approach prevents modifications from affecting + * neighbor reads within the same iteration. + * + * 3. **Deferred write-back** -- After processing Z-slice z, write the + * completed z-1 slice from the write buffer back to the data store + * using copyFromBuffer. This keeps writes sequential. + * + * 4. **Rotate windows** -- Swap both read and write buffer slots forward + * by one Z-layer. Load the next Z+1 slice. + * + * 5. **Flush remaining slices** -- After the Z-loop exits, write back the + * last slice (or the single slice for 1-deep volumes). + * + * ## Key Design Note: Why uint8 buffers? + * + * std::vector uses bit-packing, which means individual elements cannot + * be addressed by pointer. Since the rolling window needs random element + * access by index, uint8 buffers (0 or 1) are used instead. A separate + * bool[] buffer is used for the copyIntoBuffer/copyFromBuffer calls which + * require a contiguous bool array. + */ + #include "ErodeDilateMask.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -42,30 +81,41 @@ Result<> ErodeDilateMask::operator()() static_cast(udims[2]), }; + // Precompute face-neighbor index offsets and iteration order std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); - // Z-slice buffering: maintain rolling window of 3 adjacent Z-slices for mask - // to avoid random OOC chunk access during neighbor lookups. + // ---- Z-slice buffering setup ---- + // Maintain a rolling window of 3 adjacent Z-slices in memory to avoid + // random per-voxel access to the OOC data store during neighbor lookups. const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); - // Rolling window: slot 0 = z-1, slot 1 = z (current), slot 2 = z+1 + // READ buffer: maskSlices holds the original mask state for this iteration. + // Slot 0 = z-1, slot 1 = z (current), slot 2 = z+1. + // Uses uint8 (0/1) because std::vector is bit-packed and does not + // support pointer-based element access. std::array, 3> maskSlices; for(auto& ms : maskSlices) { ms.resize(sliceSize); } - // maskCopy uses same rolling window structure for output + + // WRITE buffer: maskCopySlices accumulates modifications for this iteration. + // Starts as a copy of maskSlices and is modified during the scan. std::array, 3> maskCopySlices; for(auto& ms : maskCopySlices) { ms.resize(sliceSize); } - // Temporary bool buffer for bulk I/O (std::vector is bit-packed, so use unique_ptr) + // Temporary bool[] buffer for bulk I/O with the data store. + // copyIntoBuffer/copyFromBuffer require contiguous bool arrays, + // so we convert between bool and uint8 during reads and writes. auto boolBuf = std::make_unique(sliceSize); auto& maskStore = mask.getDataStoreRef(); + // Reads one Z-slice from the data store into both the read and write buffers. + // The bool->uint8 conversion happens here. auto readMaskSlice = [&](int64 z, usize slot) { const usize zOffset = static_cast(z) * sliceSize; maskStore.copyIntoBuffer(zOffset, nonstd::span(boolBuf.get(), sliceSize)); @@ -76,23 +126,27 @@ Result<> ErodeDilateMask::operator()() } }; - // Face neighbor ordering: 0=-Z, 1=-Y, 2=-X, 3=+X, 4=+Y, 5=+Z + // Maps face-neighbor index to rolling-window slot: + // -Z -> slot 0, -Y/-X/+X/+Y -> slot 1 (same Z), +Z -> slot 2 constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + // ---- Main iteration loop ---- for(int32 iteration = 0; iteration < m_InputValues->NumIterations; iteration++) { m_MessageHandler(IFilter::Message::Type::Info, fmt::format("Iteration {}", iteration)); - // Initialize rolling window: load z=0 into slot 1, z=1 into slot 2 + // Re-initialize rolling window from the (potentially modified) store. + // z=0 -> slot 1 (current), z=1 -> slot 2 (next). readMaskSlice(0, 1); if(dims[2] > 1) { readMaskSlice(1, 2); } + // ---- Z-slice scan loop ---- for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - // Advance rolling window for z > 0 + // Advance both read and write rolling windows for z > 0 if(zIdx > 0) { std::swap(maskSlices[0], maskSlices[1]); @@ -105,14 +159,18 @@ Result<> ErodeDilateMask::operator()() } } + // ---- Inner XY scan ---- for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) { const usize inSlice = static_cast(yIdx * dims[0] + xIdx); + // Only process false (unmasked) voxels -- these are the boundary + // candidates for both dilation and erosion. if(maskSlices[1][inSlice] == 0) { + // Determine valid face neighbors and mask off user-disabled directions std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); if(!m_InputValues->XDirOn) { @@ -130,6 +188,8 @@ Result<> ErodeDilateMask::operator()() isValidFaceNeighbor[k_PositiveZNeighbor] = false; } + // Map each face neighbor to its in-slice offset within the + // appropriate rolling-window slot. const std::array neighborInSlice = { inSlice, // -Z: same xy position in prev slice static_cast((yIdx - 1) * dims[0] + xIdx), // -Y @@ -148,10 +208,17 @@ Result<> ErodeDilateMask::operator()() if(m_InputValues->Operation == detail::k_DilateIndex && maskSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] != 0) { + // DILATION: This false voxel has a true neighbor, so it + // should become true. Write into the copy buffer for the + // current slice (slot 1). maskCopySlices[1][inSlice] = 1; } if(m_InputValues->Operation == detail::k_ErodeIndex && maskSlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] != 0) { + // EROSION: This false voxel has a true neighbor. Set the + // neighbor to false in the copy buffer. The neighbor may + // be in a different Z-slice (slot 0 or 2), which is why + // we write to maskCopySlices[k_NeighborSlot[faceIndex]]. maskCopySlices[k_NeighborSlot[faceIndex]][neighborInSlice[faceIndex]] = 0; } } @@ -159,7 +226,10 @@ Result<> ErodeDilateMask::operator()() } } - // Write back the completed z-1 slice using bulk I/O + // ---- Write back the completed z-1 slice using bulk I/O ---- + // After processing slice z, the modifications for z-1 are finalized + // (no future voxel at z+1 or beyond can affect z-1's write buffer). + // Convert uint8 -> bool and write back via copyFromBuffer. if(zIdx > 0) { const usize prevZOffset = static_cast(zIdx - 1) * sliceSize; @@ -171,9 +241,10 @@ Result<> ErodeDilateMask::operator()() } } - // Write back the last slice(s) using bulk I/O + // ---- Flush the last (or only) Z-slice ---- if(dims[2] == 1) { + // Single-slice volume: the current slot is still in position 1 for(usize i = 0; i < sliceSize; i++) { boolBuf[i] = (maskCopySlices[1][i] != 0); @@ -182,6 +253,8 @@ Result<> ErodeDilateMask::operator()() } else { + // Multi-slice volume: after the loop, the last slice ended up in + // slot 1 (due to swaps), which corresponds to dims[2]-1. const usize lastZOffset = static_cast(dims[2] - 1) * sliceSize; for(usize i = 0; i < sliceSize; i++) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp index 1b7ac7bc6e..e5703265be 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ErodeDilateMask.hpp @@ -25,21 +25,65 @@ static inline constexpr ChoicesParameter::ValueType k_ErodeIndex = 1ULL; /** * @struct ErodeDilateMaskInputValues * @brief Holds all user-supplied parameters for the ErodeDilateMask algorithm. + * + * Populated by the filter's preflight/execute methods and passed into the + * algorithm to decouple it from the parameter system. */ struct SIMPLNXCORE_EXPORT ErodeDilateMaskInputValues { - ChoicesParameter::ValueType Operation; - int32 NumIterations; - bool XDirOn; - bool YDirOn; - bool ZDirOn; - DataPath MaskArrayPath; - DataPath InputImageGeometry; + ChoicesParameter::ValueType Operation; ///< Morphological operation: detail::k_DilateIndex (0) or detail::k_ErodeIndex (1) + int32 NumIterations; ///< Number of erosion/dilation passes to perform + bool XDirOn; ///< Whether to consider neighbors along the X axis + bool YDirOn; ///< Whether to consider neighbors along the Y axis + bool ZDirOn; ///< Whether to consider neighbors along the Z axis + DataPath MaskArrayPath; ///< Path to the boolean mask cell array to erode/dilate + DataPath InputImageGeometry; ///< Path to the ImageGeom that defines the voxel grid }; /** * @class ErodeDilateMask - * @brief Erodes or dilates a boolean mask array using face-neighbor connectivity. + * @brief Iterative morphological erosion or dilation of a boolean mask array + * using face-neighbor connectivity, optimized for out-of-core (OOC) + * data stores. + * + * ## Algorithm Overview + * + * This algorithm performs morphological erosion or dilation on a boolean mask + * array (true/false per voxel) rather than on FeatureIds. Unlike the + * ErodeDilateBadData algorithm, this one operates directly on the mask and + * does not propagate changes to sibling data arrays. + * + * - **Dilation**: For every false (unmasked) voxel, if any face neighbor is + * true (masked), the voxel is set to true. This grows the masked region + * outward by one cell per iteration. + * - **Erosion**: For every false voxel, if any face neighbor is true, that + * neighbor is set to false. This shrinks the masked region inward by one + * cell per iteration. + * + * The operation is applied in-place to the mask array for each iteration. + * A read-then-write pattern (using separate maskSlices and maskCopySlices + * buffers) ensures that the scan within a single iteration reads the + * original state while accumulating modifications into the copy. + * + * ## OOC Optimization Strategy + * + * 1. **3-slice rolling window (dual buffers)**: Two sets of three Z-slice + * buffers are maintained: `maskSlices` for reading the original mask state + * and `maskCopySlices` for accumulating the modified state. This dual-buffer + * approach ensures reads and writes do not interfere within a single + * iteration. + * + * 2. **uint8 intermediary for bool**: Because std::vector uses + * bit-packing (which prevents taking element addresses), the rolling + * window uses uint8 buffers. A separate bool[] buffer handles bulk I/O + * with the data store's copyIntoBuffer/copyFromBuffer API. + * + * 3. **Deferred sequential writes**: After processing each Z-slice, the + * completed z-1 slice is written back to the store using copyFromBuffer. + * This keeps writes sequential and aligned with OOC chunk boundaries. + * + * 4. **Per-iteration re-read**: Each iteration re-loads the rolling window + * from the store because the previous iteration's writes changed the mask. */ class SIMPLNXCORE_EXPORT ErodeDilateMask { @@ -65,17 +109,27 @@ class SIMPLNXCORE_EXPORT ErodeDilateMask /** * @brief Executes the erode/dilate mask algorithm. + * + * Runs NumIterations passes of the selected morphological operation over + * the entire ImageGeom volume. Each pass uses a dual-buffered 3-slice + * rolling window (read from maskSlices, write into maskCopySlices) and + * deferred sequential writes back to the data store. + * * @return Result<> indicating success or any errors encountered during execution */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag. + * @return const reference to the atomic cancellation flag + */ const std::atomic_bool& getCancel() const; private: - DataStructure& m_DataStructure; - const ErodeDilateMaskInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure holding all arrays + const ErodeDilateMaskInputValues* m_InputValues = nullptr; ///< User-supplied algorithm parameters + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked between iterations + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress/status messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp index e9f4521ff0..ca41a4941a 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.cpp @@ -1,3 +1,23 @@ +// ----------------------------------------------------------------------------- +// FillBadData.cpp -- Algorithm dispatcher for the FillBadData filter +// ----------------------------------------------------------------------------- +// +// This file contains only the dispatch logic. It inspects the storage type of +// the FeatureIds array and delegates to one of two algorithm implementations: +// +// - FillBadDataBFS: BFS flood-fill, optimal for in-core (contiguous memory) +// - FillBadDataCCL: Scanline CCL with Union-Find, optimal for out-of-core +// (chunked HDF5 storage on disk) +// +// The dispatch is performed by DispatchAlgorithm(), which checks +// the data store type of each array in the initializer list. If any array +// uses OOC storage (or if the test override ForceOocAlgorithm() is set), +// the CCL variant is selected. Otherwise, the BFS variant is used. +// +// Both algorithm variants produce identical results for the same inputs. +// The only difference is their data access pattern and memory strategy. +// ----------------------------------------------------------------------------- + #include "FillBadData.hpp" #include "FillBadDataBFS.hpp" @@ -21,8 +41,21 @@ FillBadData::FillBadData(DataStructure& dataStructure, const IFilter::MessageHan FillBadData::~FillBadData() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to either BFS or CCL based on the FeatureIds array's storage type. + * + * The FeatureIds array is the primary input: it is read during region discovery + * and written during the fill phase. If it uses OOC storage, the CCL algorithm + * is selected because BFS flood-fill would trigger random chunk accesses that + * cause severe performance degradation (chunk thrashing). The CCL algorithm + * processes data in strictly sequential Z-slice order, which aligns with the + * chunked storage layout and avoids thrashing. + */ Result<> FillBadData::operator()() { + // Check the FeatureIds array to determine storage type (in-core vs OOC). + // This single array drives the dispatch decision because it is the most + // heavily accessed array during both region discovery and fill phases. auto* featureIdsArray = m_DataStructure.getDataAs(m_InputValues->featureIdsArrayPath); return DispatchAlgorithm({featureIdsArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp index db21d91419..1f84e5538c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadData.hpp @@ -11,19 +11,45 @@ namespace nx::core { +/** + * @struct FillBadDataInputValues + * @brief Holds all user-specified parameters for the FillBadData algorithm. + * + * This struct is populated by FillBadDataFilter and passed to the algorithm + * dispatcher. It is shared between the BFS and CCL algorithm variants. + */ struct SIMPLNXCORE_EXPORT FillBadDataInputValues { - int32 minAllowedDefectSizeValue; - bool storeAsNewPhase; - DataPath featureIdsArrayPath; - DataPath cellPhasesArrayPath; - std::vector ignoredDataArrayPaths; - DataPath inputImageGeometry; + int32 minAllowedDefectSizeValue; ///< Minimum voxel count for a bad-data region to be preserved as a large defect (regions smaller than this are filled) + bool storeAsNewPhase; ///< If true, large defect regions are assigned to a new phase (maxPhase + 1) for visualization + DataPath featureIdsArrayPath; ///< Path to the cell-level FeatureIds array (int32); voxels with value 0 are "bad data" + DataPath cellPhasesArrayPath; ///< Path to the cell-level Phases array (int32); only used when storeAsNewPhase is true + std::vector ignoredDataArrayPaths; ///< Cell arrays that should NOT be updated during the fill (e.g., arrays the user wants to preserve) + DataPath inputImageGeometry; ///< Path to the ImageGeom that defines the voxel grid dimensions }; /** * @class FillBadData - * @brief Dispatcher that selects between BFS (in-core) and CCL (out-of-core) algorithms. + * @brief Dispatcher that selects between BFS (in-core) and CCL (out-of-core) algorithms + * for filling bad data regions in an image geometry. + * + * This class does not contain algorithm logic itself. It inspects the storage + * type of the FeatureIds array and delegates to one of two algorithm classes: + * + * - **FillBadDataBFS** (in-core): Uses breadth-first search (BFS) flood-fill + * with O(N) temporary buffers (neighbors array, visited flags). Efficient when + * data fits in RAM because BFS queue access is fast and random access to the + * contiguous in-memory buffer is O(1). + * + * - **FillBadDataCCL** (out-of-core): Uses a four-phase approach with scanline + * Connected Component Labeling (CCL) and Union-Find. Processes data in Z-slice + * buffers with strictly sequential access patterns. Avoids the random access + * pattern of BFS that causes catastrophic chunk load/evict cycles ("chunk + * thrashing") when data is stored on disk in compressed HDF5 chunks. + * + * The dispatch decision is made by DispatchAlgorithm(), which checks + * whether any input array uses out-of-core storage (or if the global + * ForceOocAlgorithm() test flag is set). * * @see FillBadDataBFS for the in-core-optimized implementation. * @see FillBadDataCCL for the out-of-core-optimized implementation. @@ -49,15 +75,20 @@ class SIMPLNXCORE_EXPORT FillBadData /** * @brief Dispatches to either BFS or CCL algorithm based on data residency. + * + * Checks whether the FeatureIds array uses out-of-core storage. If so (or if + * ForceOocAlgorithm() is true), constructs and runs FillBadDataCCL. Otherwise, + * constructs and runs FillBadDataBFS. Both produce identical results. + * * @return Result indicating success or an error with a descriptive message. */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const FillBadDataInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const FillBadDataInputValues* m_InputValues = nullptr; ///< Non-owning pointer to the filter parameter values + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked during long-running phases + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for emitting progress/informational messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp index 63051e4d9e..448db4a17e 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.cpp @@ -1,3 +1,26 @@ +// ----------------------------------------------------------------------------- +// FillBadDataBFS.cpp -- In-core BFS flood-fill algorithm for filling bad data +// ----------------------------------------------------------------------------- +// +// This file implements the BFS (breadth-first search) variant of the FillBadData +// algorithm, optimized for in-core (contiguous memory) data access. The algorithm +// identifies connected regions of bad data (FeatureId == 0), classifies them by +// size, and fills small regions by copying cell data from neighboring good features. +// +// The BFS approach uses O(N) temporary buffers and relies on random access to +// both the FeatureIds array and the temporary vectors. This is efficient when all +// data fits in RAM but causes catastrophic chunk thrashing when data is stored +// out-of-core in compressed HDF5 chunks. For OOC data, FillBadDataCCL should be +// used instead (selected automatically by the FillBadData dispatcher). +// +// Algorithm Steps: +// Step 1: Linear scan to find max FeatureId (and optionally max Phase) +// Step 2: BFS flood-fill to discover and classify connected bad-data regions +// Step 3: Iterative morphological dilation to fill small regions via neighbor voting +// +// See FillBadDataBFS.hpp for detailed algorithm documentation. +// ----------------------------------------------------------------------------- + #include "FillBadDataBFS.hpp" #include "FillBadData.hpp" @@ -29,6 +52,14 @@ namespace // // All components of the tuple are copied (e.g., 3-component RGB, 6-component // tensor, etc.), preserving multi-component array semantics. +// +// WHY featureIds are checked during copy: +// The iterative fill processes one dilation layer at a time. Within a single +// iteration, a voxel that was just filled (featureId changed from -1 to a +// positive value) must NOT serve as a copy source for other voxels in the same +// iteration, because its non-featureId arrays have not yet been updated. The +// check `featureIds[neighbor] > 0` combined with updating featureIds LAST +// (after all other arrays) ensures this ordering is maintained. // ----------------------------------------------------------------------------- template void FillBadDataUpdateTuples(const Int32AbstractDataStore& featureIds, AbstractDataStore& outputDataStore, const std::vector& neighbors) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp index 6291ce0e3c..dd5a5b0d0c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataBFS.hpp @@ -13,15 +13,66 @@ struct FillBadDataInputValues; /** * @class FillBadDataBFS - * @brief BFS flood-fill algorithm for filling bad data regions. + * @brief In-core BFS flood-fill algorithm for filling bad data regions in an + * image geometry. * - * This is the in-core-optimized implementation. It uses BFS (breadth-first search) - * to identify connected components of bad data, then iteratively fills small regions - * by voting among face neighbors. Uses O(N) temporary buffers (neighbors, alreadyChecked) - * which is efficient when data fits in RAM. + * This is the in-core-optimized implementation of the FillBadData algorithm. + * It uses breadth-first search (BFS) to discover connected components of bad + * data (voxels with FeatureId == 0), classifies them by size, and iteratively + * fills small regions by copying cell data from the best neighboring good feature + * determined by majority vote. + * + * ## Algorithm Overview + * + * The algorithm proceeds in three steps: + * + * **Step 1 -- Scan for maximum feature ID (and optionally maximum phase).** + * A linear scan finds the largest FeatureId value, which is used to size the + * vote counter array in Step 3. + * + * **Step 2 -- BFS flood-fill to classify bad-data regions.** + * Starting from each unvisited voxel with FeatureId == 0, BFS expands to all + * face-adjacent bad-data neighbors. The resulting connected component is then + * classified by voxel count: + * - **Large regions** (>= minAllowedDefectSize): Kept as FeatureId = 0 (voids). + * Optionally assigned to a new phase (maxPhase + 1) for visualization. + * - **Small regions** (< minAllowedDefectSize): Marked with FeatureId = -1, + * indicating they should be filled in Step 3. + * + * **Step 3 -- Iterative morphological dilation (fill).** + * Each iteration scans all voxels with FeatureId == -1. For each, the 6 + * face-adjacent neighbors are tallied by feature ID (majority vote). The + * neighbor with the highest vote count becomes the "source" for that voxel. + * After the vote scan, all cell data arrays are updated by copying every + * component from the source neighbor to the target voxel. FeatureIds are + * updated LAST to prevent a freshly filled voxel from becoming a vote source + * before its other arrays are copied. Iterations repeat until no -1 voxels + * remain (the good-data boundary dilates inward by one voxel layer per + * iteration). + * + * ## Memory Usage + * + * This algorithm allocates O(N) temporary buffers: + * - `neighbors`: int32 per voxel, mapping each voxel to its best source neighbor + * - `alreadyChecked`: 1 bit per voxel (std::vector), tracking BFS visited state + * - `featureNumber`: int32 per feature (O(numFeatures)), used as a vote counter + * + * These O(N) allocations are efficient when data fits in RAM because all accesses + * are to contiguous in-memory arrays with O(1) random access. + * + * ## Why This is Not Suitable for Out-of-Core + * + * BFS expands outward from a seed in a wavefront pattern, visiting neighbors in + * an unpredictable order relative to the on-disk chunk layout. When data is stored + * in compressed HDF5 chunks, each neighbor access may trigger a chunk load/evict + * cycle. For a 300x300x300 volume stored in 64x64x64 chunks, a single BFS that + * spans the volume boundary crosses chunk boundaries thousands of times, causing + * catastrophic I/O amplification ("chunk thrashing"). The CCL variant avoids this + * by processing data in strict Z-slice sequential order. * * @see FillBadDataCCL for the out-of-core-optimized alternative. - * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + * @see FillBadData for the dispatcher that selects between BFS and CCL. + * @see AlgorithmDispatch.hpp for the dispatch mechanism. */ class SIMPLNXCORE_EXPORT FillBadDataBFS { @@ -43,15 +94,20 @@ class SIMPLNXCORE_EXPORT FillBadDataBFS /** * @brief Executes the BFS flood-fill algorithm to identify and fill bad data regions. + * + * This is the main entry point. It performs all three steps (scan, BFS classify, + * iterative fill) sequentially in a single call. The algorithm modifies the + * FeatureIds array and all non-ignored cell data arrays in place. + * * @return Result indicating success or an error with a descriptive message. */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const FillBadDataInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const FillBadDataInputValues* m_InputValues = nullptr; ///< Non-owning pointer to filter parameter values + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked during long-running loops + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for emitting progress/informational messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp index 5f9f63b29c..98e20a3a30 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.cpp @@ -1,3 +1,63 @@ +// ----------------------------------------------------------------------------- +// FillBadDataCCL.cpp -- Out-of-core CCL algorithm for filling bad data +// ----------------------------------------------------------------------------- +// +// This file implements the out-of-core optimized variant of the FillBadData +// algorithm. It replaces the BFS flood-fill approach (see FillBadDataBFS.cpp) +// with a four-phase pipeline built on scanline Connected Component Labeling +// (CCL) and Union-Find, designed to process data in strict Z-slice sequential +// order to avoid chunk thrashing in OOC storage. +// +// ## The Chunk Thrashing Problem +// +// When data is stored in compressed HDF5 chunks (e.g., 64x64x64 voxels per +// chunk), each random access to a voxel may trigger decompression of an entire +// chunk. BFS flood-fill visits neighbors in a wavefront pattern that crosses +// chunk boundaries unpredictably, causing the same chunks to be loaded and +// evicted thousands of times. For a 300x300x300 volume, this can turn a +// ~1-second in-core operation into a multi-hour ordeal. +// +// ## How CCL Avoids Thrashing +// +// The CCL approach processes voxels in a fixed Z-Y-X scan order, reading each +// Z-slice exactly once via bulk copyIntoBuffer/copyFromBuffer calls. Cross-slice +// connectivity is resolved symbolically through a Union-Find data structure, +// requiring only O(labels) memory rather than O(volume). A rolling 2-slice +// label buffer provides backward neighbor lookups using O(dimX * dimY) memory. +// +// ## Four-Phase Pipeline +// +// Phase 1: Z-Slice Sequential CCL +// Scans Z-slices sequentially, assigning provisional labels to bad-data voxels +// (FeatureId == 0). Uses a 2-slice rolling label buffer for backward neighbor +// reads. Records equivalences in Union-Find. Accumulates per-label voxel counts. +// Writes provisional labels to the FeatureIds store for Phase 3 to read. +// +// Phase 2: Global Resolution +// Flattens the Union-Find so every label points directly to its root. +// Accumulates per-label sizes to root labels for size classification. +// +// Phase 3: Region Classification and Relabeling +// Reads provisional labels from FeatureIds (one Z-slice at a time), resolves +// each to its root, and classifies by total component size: +// - Small regions (< threshold): relabeled to -1 for filling in Phase 4 +// - Large regions (>= threshold): relabeled to 0 (optionally new phase) +// +// Phase 4: Iterative Morphological Fill (Temp-File Deferred) +// Each iteration has two passes: +// - Pass 1 (Vote): 3-slice rolling window scan. For each -1 voxel, majority +// vote among face neighbors. Write (dest, src) pairs to temp file. +// - Pass 2 (Apply): Read pairs back, apply fills via 3-slice buffered bulk I/O. +// No O(N) memory allocations. Uses O(features) vote counter + temp file I/O. +// Repeats until no -1 voxels remain. +// +// ## Result Equivalence +// +// This algorithm produces identical results to FillBadDataBFS for the same inputs. +// The four-phase decomposition is purely an optimization of the data access pattern. +// +// ----------------------------------------------------------------------------- + #include "FillBadDataCCL.hpp" #include "FillBadData.hpp" @@ -14,39 +74,15 @@ using namespace nx::core; -// ----------------------------------------------------------------------------- -// FillBadData Algorithm Overview -// ----------------------------------------------------------------------------- -// -// This file implements an optimized algorithm for filling bad data (voxels with -// FeatureId == 0) in image geometries. The algorithm handles out-of-core datasets -// efficiently by processing data in Z-slice buffers and uses a four-phase approach: -// -// Phase 1: Z-Slice Sequential Connected Component Labeling (CCL) -// - Process Z-slices sequentially, assigning provisional labels to bad data regions -// - Use Union-Find to track equivalences between labels across slice boundaries -// - Track size of each connected component -// -// Phase 2: Global Resolution -// - Flatten Union-Find structure to resolve all equivalences -// - Accumulate region sizes to root labels -// -// Phase 3: Region Classification and Relabeling -// - Classify regions as "small" (below threshold) or "large" (above threshold) -// - Small regions: mark with -1 for filling in Phase 4 -// - Large regions: keep as 0 or assign to new phase (if requested) -// -// Phase 4: Iterative Morphological Fill (On-Disk Deferred) -// - Uses a temporary file to defer fills: Pass 1 writes (dest, src) pairs, -// Pass 2 reads them back and applies fills. -// - No O(N) memory allocations — uses O(features) vote counters + temp file I/O. -// -// ----------------------------------------------------------------------------- - namespace { // ----------------------------------------------------------------------------- -// Helper: Copy all components of a single tuple from src to dest in a data store. +// copyTuple -- Copy all components of a single tuple from src to dest in a data store. +// ----------------------------------------------------------------------------- +// This is used as a fallback for individual tuple copies when slice-buffered +// bulk I/O is not possible (e.g., for std::vector which lacks .data()). +// For non-bool types, the SliceBufferedCopyFunctor below is preferred because +// it amortizes I/O across many tuples in the same Z-slice. // ----------------------------------------------------------------------------- template void copyTuple(AbstractDataStore& store, int64 dest, int64 src) @@ -57,7 +93,13 @@ void copyTuple(AbstractDataStore& store, int64 dest, int64 src) store.copyFromBuffer(static_cast(dest) * numComp, nonstd::span(buffer.get(), numComp)); } -// Functor for type-dispatched single-tuple copy +/** + * @brief Type-dispatched functor for copying a single tuple between indices. + * + * Used as a fallback when slice-buffered bulk I/O is not possible. In the + * Phase 4 fill pipeline, this is only used for bool arrays (extremely rare + * in practice) where std::vector prevents pointer-based bulk access. + */ struct CopyTupleFunctor { template @@ -68,8 +110,30 @@ struct CopyTupleFunctor } }; -// Functor for type-dispatched slice-buffered copy of all pairs for one array. -// Uses a 3-slice rolling window to apply fills with bulk I/O. +// ----------------------------------------------------------------------------- +// SliceBufferedCopyFunctor +// ----------------------------------------------------------------------------- +// Type-dispatched functor for applying fill pairs to a single cell data array +// using a 3-slice rolling window for bulk I/O. This is the core I/O optimization +// in Phase 4 that makes the CCL variant OOC-friendly. +// +// WHY a 3-slice window: +// Each fill pair (dest, src) copies data from a source voxel to a destination +// voxel. The source is always a face-adjacent neighbor of the destination, so +// it can be in the same Z-slice, the previous Z-slice (z-1), or the next Z-slice +// (z+1). By keeping three consecutive Z-slices in memory [prev | cur | next], +// we can resolve any fill pair without additional I/O. Since pairs are generated +// in Z-Y-X order (from the Phase 4 vote scan), consecutive pairs tend to be in +// the same Z-slice, and the window only shifts when the destination moves to a +// new Z-slice. +// +// I/O REDUCTION: +// Without this optimization, each fill pair would require two per-tuple OOC +// accesses (one read for src, one write for dest), resulting in potentially +// millions of individual chunk decompressions. With the 3-slice window, each +// Z-slice is read/written at most once per iteration, reducing I/O to +// approximately 3 * dimZ bulk reads + dimZ bulk writes per array per iteration. +// ----------------------------------------------------------------------------- struct SliceBufferedCopyFunctor { template @@ -163,7 +227,7 @@ struct TempFileGuard } // namespace // ----------------------------------------------------------------------------- -// FillBadData Implementation +// FillBadDataCCL Implementation // ----------------------------------------------------------------------------- FillBadDataCCL::FillBadDataCCL(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const FillBadDataInputValues* inputValues) @@ -187,14 +251,37 @@ const std::atomic_bool& FillBadDataCCL::getCancel() const // PHASE 1: Z-Slice Sequential Connected Component Labeling (CCL) // ----------------------------------------------------------------------------- // -// Performs connected component labeling on bad data voxels (FeatureId == 0) -// using a Z-slice sequential scanline algorithm. Uses positive labels and an -// in-memory provisional labels buffer to avoid cross-slice OOC reads. +// This phase performs connected component labeling on bad-data voxels +// (FeatureId == 0) using a Z-slice sequential scanline algorithm. The key +// insight that makes this OOC-friendly is that scanline CCL only needs to +// check three BACKWARD neighbors (x-1, y-1, z-1), all of which have already +// been processed. This means we can process voxels in strict Z-Y-X order, +// reading each Z-slice exactly once with a bulk copyIntoBuffer call. +// +// Data structures: +// - labelBuffer: Rolling 2-slice buffer (2 * dimX * dimY int32 values). +// Alternates between even/odd Z indices via (z % 2) to store provisional +// labels for the current and previous Z-slice. This provides O(1) backward +// neighbor lookups without storing labels for the entire volume. +// - unionFind: Tracks equivalences between provisional labels assigned to +// different parts of the same connected component. When a voxel has multiple +// differently-labeled backward neighbors, their labels are united. +// - featureIdsSlice: Temporary buffer for reading/writing one Z-slice of +// FeatureIds. Provisional labels are written back to the FeatureIds store +// so that Phase 3 can read them without needing a separate label volume. // -// @param featureIdsStore The feature IDs data store (maybe out-of-core) -// @param unionFind Union-Find structure for tracking label equivalences -// @param nextLabel Next label to assign (incremented as new labels are created) -// @param dims Image dimensions [X, Y, Z] +// Label assignment: +// - Each bad-data voxel with no labeled backward neighbor gets a new +// provisional label (nextLabel++). +// - Each bad-data voxel with one or more labeled backward neighbors inherits +// the smallest label. If multiple differently-labeled neighbors exist, +// they are united in the Union-Find. +// - Good-data voxels (FeatureId != 0) are skipped and retain their original +// FeatureId values. +// +// Provisional labels start at (maxExistingFeatureId + 1) to avoid collision +// with existing good-feature IDs. This allows Phase 3 to distinguish between +// original feature IDs and CCL-assigned labels using a simple threshold check. // ----------------------------------------------------------------------------- void FillBadDataCCL::phaseOneCCL(Int32AbstractDataStore& featureIdsStore, UnionFind& unionFind, int32& nextLabel, const std::array& dims) { @@ -745,7 +832,13 @@ Result<> FillBadDataCCL::phaseFourIterativeFill(Int32AbstractDataStore& featureI } // ----------------------------------------------------------------------------- -// Main Algorithm Entry Point +// Main Algorithm Entry Point -- Orchestrates Phases 1-4 +// ----------------------------------------------------------------------------- +// This method performs the initial setup (finding max feature ID and max phase +// via chunked bulk scans), initializes the Union-Find, and then calls each +// phase method sequentially. The chunked scans use a Z-slice-sized buffer +// (dimX * dimY) to read feature IDs and phases in bulk, avoiding per-element +// OOC access during the setup phase. // ----------------------------------------------------------------------------- Result<> FillBadDataCCL::operator()() { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp index 4822e180a7..98c793702f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FillBadDataCCL.hpp @@ -23,13 +23,77 @@ struct FillBadDataInputValues; /** * @class FillBadDataCCL - * @brief CCL-based algorithm for filling bad data regions, optimized for out-of-core. + * @brief Out-of-core optimized algorithm for filling bad data regions using + * scanline Connected Component Labeling (CCL) with Union-Find. * - * Uses chunk-sequential connected component labeling with a 2-slice rolling buffer - * to avoid O(N) memory allocations. Designed for datasets that may exceed available RAM. + * This is the OOC-optimized implementation of the FillBadData algorithm. It + * replaces the BFS flood-fill approach with a four-phase pipeline that processes + * data in strict Z-slice sequential order, avoiding the random access pattern + * that causes chunk thrashing when data is stored in compressed HDF5 chunks. + * + * ## Why CCL Instead of BFS for Out-of-Core + * + * BFS flood-fill expands outward from a seed voxel, visiting neighbors in a + * wavefront pattern. This wavefront crosses Z-slice (and therefore chunk) + * boundaries unpredictably, causing the HDF5 chunk cache to repeatedly load + * and evict the same chunks. For a typical 300x300x300 volume in 64x64x64 + * chunks, a single BFS can trigger millions of chunk I/O operations. + * + * CCL avoids this by scanning voxels in a fixed Z-Y-X order, only looking at + * backward neighbors (x-1, y-1, z-1). This means each Z-slice is read exactly + * once during the forward scan. Cross-slice connectivity is tracked via a + * Union-Find data structure (O(labels) memory), and a rolling 2-slice label + * buffer provides backward neighbor lookups using only O(dimX * dimY) memory + * instead of O(volume). + * + * ## Four-Phase Algorithm + * + * **Phase 1 -- Scanline CCL (Z-slice sequential):** + * Processes one Z-slice at a time using copyIntoBuffer/copyFromBuffer for bulk + * I/O. For each bad-data voxel (FeatureId == 0), checks three backward neighbors + * (x-1, y-1, z-1) in a rolling 2-slice label buffer. Assigns provisional labels + * and records equivalences in a UnionFind structure. Writes provisional labels + * back to the FeatureIds store for use in Phase 3. Accumulates per-label voxel + * counts for size classification. + * + * **Phase 2 -- Global resolution:** + * Flattens the UnionFind structure so every label points directly to its root, + * and accumulates per-label sizes to root labels. After this phase, querying + * the size of any label gives the total voxel count of its connected component. + * + * **Phase 3 -- Region classification and relabeling:** + * Reads provisional labels back from the FeatureIds store (one Z-slice at a + * time) and classifies each component: + * - Small regions (< minAllowedDefectSize): Relabeled to -1 for filling + * - Large regions (>= minAllowedDefectSize): Relabeled to 0 (optionally + * assigned to a new phase for visualization) + * Original good-feature labels (in [1, startLabel)) are left unchanged. + * + * **Phase 4 -- Iterative morphological fill (temp-file deferred):** + * Each iteration has two passes: + * - Pass 1 (Vote): Scans voxels using a 3-slice rolling window. For each -1 + * voxel, finds the best positive-FeatureId neighbor via majority vote. Writes + * (dest, src) index pairs to a temporary file. FeatureIds are read-only during + * this pass, ensuring all votes see the pre-iteration state. + * - Pass 2 (Apply): Reads pairs back from the temp file and applies fills using + * a 3-slice rolling buffer per cell array, converting per-tuple random accesses + * into bulk slice I/O operations. + * Iterations repeat until no -1 voxels remain. + * + * ## Memory Usage + * + * - Phase 1: O(dimX * dimY) for the 2-slice label buffer + O(labels) for UnionFind + * - Phase 2: O(labels) for flatten + * - Phase 3: O(dimX * dimY) for slice buffers + O(labels) for classification + * - Phase 4: O(numFeatures) for vote counter + O(dimX * dimY) for 3-slice windows + * + temp file I/O for deferred fill pairs + * + * No O(N) memory allocations are made at any point (where N = total voxels). * * @see FillBadDataBFS for the in-core-optimized alternative. - * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + * @see FillBadData for the dispatcher that selects between BFS and CCL. + * @see AlgorithmDispatch.hpp for the dispatch mechanism. + * @see UnionFind for the disjoint-set data structure used in Phases 1-3. */ class SIMPLNXCORE_EXPORT FillBadDataCCL { @@ -50,7 +114,12 @@ class SIMPLNXCORE_EXPORT FillBadDataCCL FillBadDataCCL& operator=(FillBadDataCCL&&) noexcept = delete; /** - * @brief Executes the CCL-based algorithm to identify and fill bad data regions. + * @brief Executes the four-phase CCL algorithm to identify and fill bad data regions. + * + * Orchestrates Phases 1-4 sequentially, emitting progress messages between + * phases. The algorithm modifies the FeatureIds array and all non-ignored cell + * data arrays in place, producing results identical to FillBadDataBFS. + * * @return Result indicating success or an error with a descriptive message. */ Result<> operator()(); @@ -62,15 +131,68 @@ class SIMPLNXCORE_EXPORT FillBadDataCCL const std::atomic_bool& getCancel() const; private: + /** + * @brief Phase 1: Z-slice sequential scanline CCL for bad-data voxels. + * + * Scans the volume one Z-slice at a time, assigning provisional labels to + * bad-data voxels (FeatureId == 0) and recording equivalences in the + * Union-Find structure. Uses a rolling 2-slice label buffer for backward + * neighbor lookups (O(dimX * dimY) memory). Writes provisional labels back + * to the FeatureIds store for Phase 3 to read. + * + * @param featureIdsStore The FeatureIds data store (potentially out-of-core). + * @param unionFind Union-Find structure for tracking label equivalences. + * @param nextLabel Next label to assign; incremented as new labels are created. + * @param dims Image dimensions [X, Y, Z]. + */ static void phaseOneCCL(Int32AbstractDataStore& featureIdsStore, UnionFind& unionFind, int32& nextLabel, const std::array& dims); + + /** + * @brief Phase 2: Flatten the Union-Find to resolve all equivalences. + * + * After this call, every label points directly to its root and all per-label + * sizes are accumulated at root labels. Subsequent find() calls are O(1). + * + * @param unionFind Union-Find structure to flatten. + */ static void phaseTwoGlobalResolution(UnionFind& unionFind); + + /** + * @brief Phase 3: Classify regions by size and relabel the FeatureIds store. + * + * Reads provisional labels (written during Phase 1) back from the FeatureIds + * store one Z-slice at a time. Each CCL label is resolved to its root and + * classified: small regions become -1 (for filling), large regions become 0 + * (optionally assigned to a new phase). + * + * @param featureIdsStore The FeatureIds data store to relabel. + * @param cellPhasesPtr Optional cell phases array (used when storeAsNewPhase is true). + * @param startLabel First provisional CCL label (= maxExistingFeatureId + 1). + * @param nextLabel One past the last provisional CCL label assigned. + * @param unionFind Flattened Union-Find for root resolution. + * @param maxPhase Maximum existing phase value (large regions get maxPhase + 1). + */ void phaseThreeRelabeling(Int32AbstractDataStore& featureIdsStore, Int32Array* cellPhasesPtr, int32 startLabel, int32 nextLabel, UnionFind& unionFind, usize maxPhase) const; + + /** + * @brief Phase 4: Iterative morphological fill using temp-file deferred I/O. + * + * Each iteration scans voxels via a 3-slice rolling window (Pass 1), performs + * majority voting among face neighbors, and writes (dest, src) pairs to a + * temporary file. Pass 2 reads pairs back and applies fills using slice-buffered + * bulk I/O. Repeats until no -1 voxels remain. + * + * @param featureIdsStore The FeatureIds data store to fill. + * @param dims Image dimensions [X, Y, Z]. + * @param numFeatures Maximum feature ID (used to size the vote counter). + * @return Result indicating success or an error (e.g., temp file creation failure). + */ Result<> phaseFourIterativeFill(Int32AbstractDataStore& featureIdsStore, const std::array& dims, usize numFeatures) const; - DataStructure& m_DataStructure; - const FillBadDataInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const FillBadDataInputValues* m_InputValues = nullptr; ///< Non-owning pointer to filter parameter values + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked during long-running phases + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for emitting progress/informational messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp index 624d7b60b2..22029e9a28 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.cpp @@ -1,3 +1,22 @@ +// ----------------------------------------------------------------------------- +// IdentifySample.cpp -- Algorithm dispatcher for the IdentifySample filter +// ----------------------------------------------------------------------------- +// +// This file contains only the dispatch logic. It inspects the storage type of +// the mask array and delegates to one of two algorithm implementations: +// +// - IdentifySampleBFS: BFS flood-fill, optimal for in-core (contiguous memory) +// - IdentifySampleCCL: Scanline CCL with Union-Find, optimal for out-of-core +// (chunked HDF5 storage on disk) +// +// When slice-by-slice mode is enabled, both variants delegate to the same +// shared IdentifySampleSliceBySliceFunctor (in IdentifySampleCommon.hpp), +// which uses BFS on individual 2D slices. Slice-by-slice BFS is always safe +// because a single slice fits in memory regardless of OOC storage. +// +// Both algorithm variants produce identical results for the same inputs. +// ----------------------------------------------------------------------------- + #include "IdentifySample.hpp" #include "IdentifySampleBFS.hpp" @@ -21,8 +40,22 @@ IdentifySample::IdentifySample(DataStructure& dataStructure, const IFilter::Mess IdentifySample::~IdentifySample() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to either BFS or CCL based on the mask array's storage type. + * + * The mask array is the sole input/output: it is read during component discovery + * and written during the masking/fill phases. If it uses OOC storage, the CCL + * algorithm is selected because BFS flood-fill would trigger random chunk accesses + * that cause severe performance degradation (chunk thrashing). The CCL algorithm + * processes data in strictly sequential Z-slice order, which aligns with the + * chunked storage layout and avoids thrashing. + */ Result<> IdentifySample::operator()() { + // Check the mask array to determine storage type (in-core vs OOC). + // This single array drives the dispatch decision because it is the only + // data array accessed during the algorithm (unlike FillBadData which also + // accesses FeatureIds and multiple cell arrays). auto* maskArray = m_DataStructure.getDataAs(m_InputValues->MaskArrayPath); return DispatchAlgorithm({maskArray}, m_DataStructure, m_MessageHandler, m_ShouldCancel, m_InputValues); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp index a24a219f25..cf88e957e1 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySample.hpp @@ -13,23 +13,67 @@ namespace nx::core { +/** + * @struct IdentifySampleInputValues + * @brief Holds all user-specified parameters for the IdentifySample algorithm. + * + * This struct is populated by IdentifySampleFilter and passed to the algorithm + * dispatcher. It is shared between the BFS and CCL algorithm variants, and + * also by the slice-by-slice functor (IdentifySampleSliceBySliceFunctor). + */ struct SIMPLNXCORE_EXPORT IdentifySampleInputValues { - BoolParameter::ValueType FillHoles; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - ArraySelectionParameter::ValueType MaskArrayPath; - BoolParameter::ValueType SliceBySlice; - ChoicesParameter::ValueType SliceBySlicePlaneIndex; + BoolParameter::ValueType FillHoles; ///< If true, interior holes (bad-data regions fully enclosed by the sample) are filled + GeometrySelectionParameter::ValueType InputImageGeometryPath; ///< Path to the ImageGeom defining the voxel grid dimensions + ArraySelectionParameter::ValueType MaskArrayPath; ///< Path to the boolean mask array (true = good/sample, false = bad/non-sample) + BoolParameter::ValueType SliceBySlice; ///< If true, process each 2D slice independently instead of the full 3D volume + ChoicesParameter::ValueType SliceBySlicePlaneIndex; ///< Which orthogonal plane to slice along: 0 = XY, 1 = XZ, 2 = YZ }; /** * @class IdentifySample - * @brief This algorithm implements support code for the IdentifySampleFilter + * @brief Dispatcher that selects between BFS (in-core) and CCL (out-of-core) algorithms + * for identifying the largest connected sample region in an image geometry. + * + * This class does not contain algorithm logic itself. It inspects the storage + * type of the mask array and delegates to one of two algorithm classes: + * + * - **IdentifySampleBFS** (in-core): Uses BFS flood-fill with O(N) temporary + * bit vectors (checked, sample) to discover connected components. Fast when + * data fits in RAM due to O(1) random access, but causes chunk thrashing in + * OOC mode because BFS visits neighbors in an unpredictable wavefront pattern. + * + * - **IdentifySampleCCL** (out-of-core): Uses scanline Connected Component + * Labeling (CCL) with a rolling 2-slice label buffer and Vector Union-Find. + * Processes data in strict Z-slice sequential order, reading each slice + * exactly once. Uses a "replay" technique to avoid O(volume) label storage: + * the forward CCL scan is re-executed deterministically to re-derive labels + * on the fly during the apply phase. + * + * Both variants support the slice-by-slice mode, which delegates to the shared + * IdentifySampleSliceBySliceFunctor (defined in IdentifySampleCommon.hpp). + * Slice-by-slice mode uses BFS on individual 2D slices, which is always safe + * because a single slice fits in memory regardless of OOC storage. + * + * The dispatch decision is made by DispatchAlgorithm(), which checks + * whether the mask array uses out-of-core storage (or if the global + * ForceOocAlgorithm() test flag is set). + * + * @see IdentifySampleBFS for the in-core-optimized implementation. + * @see IdentifySampleCCL for the out-of-core-optimized implementation. + * @see IdentifySampleCommon.hpp for the shared slice-by-slice functor. + * @see AlgorithmDispatch.hpp for the dispatch mechanism. */ - class SIMPLNXCORE_EXPORT IdentifySample { public: + /** + * @brief Constructs the dispatcher with the required context for algorithm selection. + * @param dataStructure The data structure containing the arrays to process. + * @param mesgHandler Handler for progress and informational messages. + * @param shouldCancel Cancellation flag checked during execution. + * @param inputValues Filter parameter values controlling identification behavior. + */ IdentifySample(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, IdentifySampleInputValues* inputValues); ~IdentifySample() noexcept; @@ -38,13 +82,24 @@ class SIMPLNXCORE_EXPORT IdentifySample IdentifySample& operator=(const IdentifySample&) = delete; IdentifySample& operator=(IdentifySample&&) noexcept = delete; + /** + * @brief Dispatches to either BFS or CCL algorithm based on data residency. + * + * Checks whether the mask array uses out-of-core storage. If so (or if + * ForceOocAlgorithm() is true), constructs and runs IdentifySampleCCL. + * Otherwise, constructs and runs IdentifySampleBFS. Both produce identical + * results. In slice-by-slice mode, both variants delegate to the same + * shared IdentifySampleSliceBySliceFunctor. + * + * @return Result indicating success or an error with a descriptive message. + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const IdentifySampleInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const IdentifySampleInputValues* m_InputValues = nullptr; ///< Non-owning pointer to filter parameter values + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked during long-running phases + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for emitting progress/informational messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp index 43eaa152a0..6d4a27fd21 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.cpp @@ -1,3 +1,25 @@ +// ----------------------------------------------------------------------------- +// IdentifySampleBFS.cpp -- In-core BFS flood-fill for sample identification +// ----------------------------------------------------------------------------- +// +// This file implements the BFS (breadth-first search) variant of the +// IdentifySample algorithm, optimized for in-core (contiguous memory) data +// access. The algorithm identifies the largest connected component of "good" +// voxels (mask == true) as the sample, removes satellite regions, and +// optionally fills interior holes. +// +// The BFS approach uses O(N) temporary bit vectors and relies on random access +// to the mask array via getValue(). This is efficient when data fits in RAM +// but causes chunk thrashing when data is stored out-of-core in compressed +// HDF5 chunks. For OOC data, IdentifySampleCCL should be used instead +// (selected automatically by the IdentifySample dispatcher). +// +// When slice-by-slice mode is enabled, this class delegates to the shared +// IdentifySampleSliceBySliceFunctor which performs BFS on individual 2D slices. +// +// See IdentifySampleBFS.hpp for detailed algorithm documentation. +// ----------------------------------------------------------------------------- + #include "IdentifySampleBFS.hpp" #include "IdentifySample.hpp" diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp index e506bcb158..7fce5bd945 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleBFS.hpp @@ -13,15 +13,61 @@ struct IdentifySampleInputValues; /** * @class IdentifySampleBFS - * @brief BFS flood-fill algorithm for identifying the largest sample region. + * @brief In-core BFS flood-fill algorithm for identifying the largest connected + * sample region in an image geometry. * - * This is the in-core-optimized implementation. It uses BFS (breadth-first search) - * with std::vector for tracking visited voxels, which is memory-efficient - * (1 bit per voxel) and fast when data is in contiguous memory. However, the random - * access pattern of BFS causes severe chunk thrashing in out-of-core mode. + * This is the in-core-optimized implementation of the IdentifySample algorithm. + * It uses breadth-first search (BFS) to discover connected components of "good" + * voxels (mask == true), identifies the largest component as the sample, and + * optionally fills interior holes in the sample boundary. + * + * ## Algorithm Overview + * + * The algorithm has two phases: + * + * **Phase 1 -- Find the largest connected component of good voxels.** + * BFS flood-fill is initiated from each unvisited good voxel. The BFS expands + * to all face-adjacent (6-connected) good neighbors, discovering one connected + * component per seed. The component with the most voxels is recorded as "the + * sample". After all components are found, any good voxels NOT in the largest + * component are set to false (removing satellite regions and noise). + * + * **Phase 2 -- Fill interior holes (optional, when FillHoles is true).** + * A second BFS pass runs on bad voxels (mask == false). Each connected component + * of bad voxels is discovered via BFS. During expansion, a `touchesBoundary` + * flag tracks whether any voxel in the component lies on the domain boundary + * (x/y/z == 0 or max). If the component does NOT touch the boundary, it is + * fully enclosed by the sample (an interior hole) and all its voxels are set + * to true. Boundary-touching components are external empty space and left as-is. + * + * ## Memory Usage + * + * Uses O(N) temporary memory: + * - `checked`: std::vector (1 bit per voxel), tracks BFS visited state + * - `sample`: std::vector (1 bit per voxel), marks voxels in the largest component + * - `currentVList`: std::vector, BFS queue (grows to component size) + * + * ## Why This is Not Suitable for Out-of-Core + * + * BFS expands outward from a seed in a wavefront pattern, visiting face-adjacent + * neighbors in an order determined by the queue. When the goodVoxels mask array + * is stored in compressed HDF5 chunks, each getValue() call may trigger chunk + * decompression. The wavefront crosses chunk boundaries unpredictably, causing + * the chunk cache to repeatedly load and evict the same chunks (chunk thrashing). + * For large volumes, this can degrade performance by orders of magnitude. The + * CCL variant avoids this by processing data in strict Z-slice sequential order. + * + * ## Slice-By-Slice Mode + * + * When the user enables slice-by-slice processing, this class delegates to the + * shared IdentifySampleSliceBySliceFunctor (defined in IdentifySampleCommon.hpp), + * which performs BFS on individual 2D slices. This is safe even for OOC data + * because a single 2D slice fits in memory. * * @see IdentifySampleCCL for the out-of-core-optimized alternative. - * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + * @see IdentifySample for the dispatcher that selects between BFS and CCL. + * @see IdentifySampleCommon.hpp for the shared slice-by-slice functor. + * @see AlgorithmDispatch.hpp for the dispatch mechanism. */ class SIMPLNXCORE_EXPORT IdentifySampleBFS { @@ -43,15 +89,20 @@ class SIMPLNXCORE_EXPORT IdentifySampleBFS /** * @brief Executes the BFS flood-fill algorithm to identify the largest sample region. + * + * If slice-by-slice mode is enabled, delegates to IdentifySampleSliceBySliceFunctor. + * Otherwise, runs the full 3D BFS algorithm (Phase 1 + optional Phase 2). + * The algorithm modifies the mask array in place. + * * @return Result indicating success or an error with a descriptive message. */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const IdentifySampleInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const IdentifySampleInputValues* m_InputValues = nullptr; ///< Non-owning pointer to filter parameter values + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked during BFS expansion + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for emitting progress/informational messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp index 454ef74f7b..4a79f73e6f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.cpp @@ -1,3 +1,48 @@ +// ----------------------------------------------------------------------------- +// IdentifySampleCCL.cpp -- Out-of-core CCL for sample identification +// ----------------------------------------------------------------------------- +// +// This file implements the out-of-core optimized variant of the IdentifySample +// algorithm. It replaces the BFS flood-fill approach (see IdentifySampleBFS.cpp) +// with scanline Connected Component Labeling (CCL) and a "replay" technique +// designed to process data in strict Z-slice sequential order, avoiding chunk +// thrashing in OOC storage. +// +// ## Architecture: Generic CCL + Replay +// +// The implementation is built on two generic template functions: +// +// 1. runForwardCCL(store, dims, condition, cancel) +// Performs a single forward scan through the volume in Z-Y-X order. For each +// voxel where `condition` returns true, assigns a provisional label by +// checking three backward neighbors (x-1, y-1, z-1) in a rolling 2-slice +// label buffer. Records equivalences in a VectorUnionFind. Returns a CCLResult +// containing the flattened Union-Find, per-root sizes, and the largest root. +// +// 2. replayForwardCCL(store, dims, unionFind, condition, action, cancel) +// Re-executes the exact same forward scan to re-derive the same provisional +// labels deterministically. For each labeled voxel, resolves the provisional +// label to its root via the (already flattened) Union-Find, then calls +// `action(data, inSlice, root, x, y, z)` to apply per-voxel logic. If the +// action modifies the slice data, the slice is written back via copyFromBuffer. +// +// This "run then replay" pattern avoids O(volume) label storage. The trade-off +// is reading each Z-slice twice (once during run, once during replay), but for +// OOC datasets the memory savings are critical -- the volume itself may not fit +// in RAM. +// +// ## Phase Structure +// +// The IdentifySampleCCLFunctor orchestrates up to four phases: +// Phase 1: runForwardCCL on good voxels -> find largest component +// Phase 2: replayForwardCCL on good voxels -> mask non-sample voxels +// Phase 3: runForwardCCL on bad voxels -> find hole components (if FillHoles) +// Phase 4a: replayForwardCCL on bad voxels -> identify boundary-touching roots +// Phase 4b: replayForwardCCL on bad voxels -> fill interior holes +// +// See IdentifySampleCCL.hpp for detailed algorithm documentation. +// ----------------------------------------------------------------------------- + #include "IdentifySampleCCL.hpp" #include "IdentifySample.hpp" diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp index cba07dface..362b2d77c9 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCCL.hpp @@ -13,19 +13,83 @@ struct IdentifySampleInputValues; /** * @class IdentifySampleCCL - * @brief Chunk-sequential CCL algorithm for identifying the largest sample region. + * @brief Out-of-core optimized algorithm for identifying the largest connected + * sample region using scanline Connected Component Labeling (CCL) with Union-Find. * - * This is the out-of-core-optimized implementation. It uses scanline Connected - * Component Labeling (CCL) with a union-find structure, processing data in chunk - * order to minimize disk I/O. The algorithm only accesses backward neighbors - * (-X, -Y, -Z) during labeling, ensuring sequential chunk access. + * This is the OOC-optimized implementation of the IdentifySample algorithm. It + * replaces the BFS flood-fill approach with scanline CCL that processes data in + * strict Z-slice sequential order, avoiding the random access pattern that causes + * chunk thrashing when data is stored in compressed HDF5 chunks. * - * Trade-off: Uses a std::vector label array (8 bytes per voxel) which is - * more memory than the BFS approach (1 bit per voxel), but avoids the random - * access pattern that causes chunk thrashing in OOC mode. + * ## Why CCL Instead of BFS for Out-of-Core + * + * BFS flood-fill visits neighbors in a wavefront pattern, crossing Z-slice (and + * chunk) boundaries unpredictably. Each getValue() call on an OOC mask array may + * trigger chunk decompression, and the wavefront pattern causes the same chunks + * to be loaded and evicted repeatedly. For large volumes (e.g., 500x500x500), + * this can turn a sub-second in-core operation into a multi-hour ordeal. + * + * CCL avoids this by scanning voxels in a fixed Z-Y-X order, only checking + * backward neighbors (x-1, y-1, z-1). Each Z-slice is read exactly once via + * a bulk copyIntoBuffer call. Cross-slice connectivity is resolved symbolically + * through a VectorUnionFind structure (O(labels) memory). + * + * ## Algorithm Phases + * + * The algorithm has up to four phases (Phases 3-4 only run if FillHoles is true): + * + * **Phase 1 -- Forward CCL on good voxels:** + * Runs runForwardCCL() with condition = (mask[i] == true). Discovers all + * connected components of good voxels, tracks per-component sizes, and identifies + * the largest component as "the sample". Uses a rolling 2-slice label buffer + * (O(dimX * dimY) memory) for backward neighbor lookups. + * + * **Phase 2 -- Replay CCL to mask non-sample voxels:** + * Runs replayForwardCCL() with the same good-voxel condition. This re-executes + * the exact same forward scan deterministically, re-deriving the same provisional + * labels on the fly. For each voxel whose resolved root is not the largest + * component, sets the mask to false (removing satellite regions). This avoids + * O(volume) label storage by recomputing labels instead of storing them. + * + * **Phase 3 -- Forward CCL on bad voxels (hole detection, optional):** + * Runs runForwardCCL() with condition = (mask[i] == false). Discovers connected + * components of non-sample space (exterior empty space + interior holes). + * + * **Phase 4 -- Replay CCL to fill interior holes (optional):** + * Two replay passes: (4a) identifies which bad-voxel components touch the domain + * boundary (these are exterior, not holes); (4b) fills interior holes (components + * NOT touching any boundary) by setting their mask voxels to true. + * + * ## The Replay Trick -- Avoiding O(Volume) Label Storage + * + * The key OOC technique in this algorithm is the "replay" approach. Scanline CCL + * label assignment is fully deterministic given the same scan order and condition. + * So instead of storing labels for the entire volume (O(volume) memory), we + * re-run the forward scan to re-derive the same labels on the fly. The already- + * flattened Union-Find resolves each re-derived label to its root in O(1). This + * trades an extra data read (reading each Z-slice twice) for massive memory + * savings -- critical when the volume itself does not fit in RAM. + * + * ## Memory Usage + * + * - Phase 1: O(dimX * dimY) for 2-slice label buffer + O(labels) for Union-Find + O(labels) for sizes + * - Phase 2: O(dimX * dimY) for 2-slice label buffer (Union-Find reused from Phase 1) + * - Phase 3: Same as Phase 1 (new Union-Find for bad-voxel components) + * - Phase 4: O(dimX * dimY) for 2-slice label buffer + O(labels) for boundary flags + * + * No O(volume) memory is allocated at any point. + * + * ## Slice-By-Slice Mode + * + * When the user enables slice-by-slice processing, this class delegates to the + * shared IdentifySampleSliceBySliceFunctor (defined in IdentifySampleCommon.hpp), + * which performs BFS on individual 2D slices. This is safe even for OOC data + * because a single 2D slice fits in memory. * * @see IdentifySampleBFS for the in-core-optimized alternative. - * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. + * @see IdentifySample for the dispatcher that selects between BFS and CCL. + * @see IdentifySampleCommon.hpp for the shared slice-by-slice functor and VectorUnionFind. + * @see AlgorithmDispatch.hpp for the dispatch mechanism. */ class SIMPLNXCORE_EXPORT IdentifySampleCCL { @@ -47,15 +111,21 @@ class SIMPLNXCORE_EXPORT IdentifySampleCCL /** * @brief Executes the CCL-based algorithm to identify the largest sample region. + * + * If slice-by-slice mode is enabled, delegates to IdentifySampleSliceBySliceFunctor. + * Otherwise, runs the full 3D CCL algorithm (Phases 1-2, plus Phases 3-4 if + * FillHoles is true). The algorithm modifies the mask array in place, producing + * results identical to IdentifySampleBFS. + * * @return Result indicating success or an error with a descriptive message. */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const IdentifySampleInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const IdentifySampleInputValues* m_InputValues = nullptr; ///< Non-owning pointer to filter parameter values + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked between Z-slice iterations + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for emitting progress/informational messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp index b6619899d3..22fe12a4f9 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/IdentifySampleCommon.hpp @@ -1,3 +1,29 @@ +// ----------------------------------------------------------------------------- +// IdentifySampleCommon.hpp -- Shared utilities for IdentifySample algorithms +// ----------------------------------------------------------------------------- +// +// This header contains two components shared between the BFS and CCL variants +// of the IdentifySample algorithm: +// +// 1. VectorUnionFind: A lightweight, vector-based union-find (disjoint set) +// data structure optimized for dense, sequentially-assigned label sets. +// Used by the CCL variant (IdentifySampleCCL) for tracking connected +// component equivalences during scanline labeling. Uses union-by-rank +// and path-halving for near-O(1) amortized operations. +// +// 2. IdentifySampleSliceBySliceFunctor: A type-dispatched functor that +// performs BFS-based sample identification on individual 2D slices of +// the volume. Used by BOTH algorithm classes when the user enables +// slice-by-slice mode. Since a single 2D slice always fits in memory, +// BFS is safe and efficient regardless of whether the underlying data +// store is in-core or out-of-core. +// +// The functor supports three orthogonal slice planes (XY, XZ, YZ) and +// includes a batched YZ code path that amortizes HDF5 I/O by reading +// each Z-slice once per batch of X positions (instead of once per X +// position), providing approximately 10x speedup for OOC data. +// ----------------------------------------------------------------------------- + #pragma once #include "simplnx/DataStructure/DataArray.hpp" @@ -14,10 +40,25 @@ namespace nx::core /** * @class VectorUnionFind - * @brief Vector-based union-find for dense label sets (labels 1..N). + * @brief Vector-based union-find (disjoint set) for dense, sequentially-assigned + * label sets (labels 1..N). + * + * Uses flat vectors instead of hash maps for O(1) indexed access with no hash + * overhead. Suitable for connected component labeling where labels are assigned + * sequentially starting from 1 and the maximum label count is not known in advance + * (internal storage grows dynamically). + * + * Features: + * - Union-by-rank for balanced merges (O(alpha(N)) amortized) + * - Path halving in find() for near-O(1) amortized lookups + * - Dynamic growth via makeSet() -- no need to pre-size + * - No flatten() method -- the CCL code accumulates sizes externally and + * resolves roots via find() after the forward scan completes * - * Uses flat vectors instead of hash maps for O(1) access. Suitable for - * connected component labeling where labels are assigned sequentially. + * This class is used by IdentifySampleCCL (in IdentifySampleCCL.cpp) for the + * 3D CCL algorithm. FillBadDataCCL uses the separate UnionFind class (in + * simplnx/Utilities/UnionFind.hpp) which includes built-in size tracking and + * a flatten() method. */ class VectorUnionFind { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp index e50e7e395b..a9524a2dc6 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.cpp @@ -8,6 +8,19 @@ using namespace nx::core; +// ============================================================================= +// MultiThresholdObjects — Dispatcher +// +// This file contains only the dispatch logic. The actual algorithm implementations +// live in MultiThresholdObjectsDirect.cpp (in-core) and +// MultiThresholdObjectsScanline.cpp (out-of-core). +// +// The dispatch checks the first required input array's storage type: if it uses +// chunked on-disk storage (OOC), the Scanline variant is selected. Since all +// input arrays in a threshold set come from the same Attribute Matrix, if one is +// OOC then all are OOC, so checking the first is sufficient. +// ============================================================================= + // ----------------------------------------------------------------------------- MultiThresholdObjects::MultiThresholdObjects(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues) @@ -22,10 +35,21 @@ MultiThresholdObjects::MultiThresholdObjects(DataStructure& dataStructure, const MultiThresholdObjects::~MultiThresholdObjects() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to the appropriate algorithm variant based on storage type. + * + * Checks the first input array referenced by the threshold configuration to determine + * if OOC storage is in use. All threshold input arrays come from the same Attribute + * Matrix, so if one is OOC, all are OOC. + * + * Both variants receive identical constructor arguments and produce identical output. + */ Result<> MultiThresholdObjects::operator()() { auto thresholdsObject = m_InputValues->ArrayThresholdsObject; const auto& requiredPaths = thresholdsObject.getRequiredPaths(); + // Check the first input array — since all arrays in a threshold set share the + // same Attribute Matrix, they all use the same storage type. const IDataArray* checkArray = nullptr; if(!requiredPaths.empty()) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp index 2371efc102..858dceaaa6 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjects.hpp @@ -11,42 +11,68 @@ #include "simplnx/Parameters/DataTypeParameter.hpp" #include "simplnx/Parameters/NumberParameter.hpp" -/** -* This is example code to put in the Execute Method of the filter. - MultiThresholdObjectsInputValues inputValues; - inputValues.ArrayThresholdsObject = filterArgs.value(array_thresholds_object); - inputValues.CreatedMaskType = filterArgs.value(created_mask_type); - inputValues.CustomFalseValue = filterArgs.value(custom_false_value); - inputValues.CustomTrueValue = filterArgs.value(custom_true_value); - inputValues.OutputDataArrayName = filterArgs.value(output_data_array_name); - inputValues.UseCustomFalseValue = filterArgs.value(use_custom_false_value); - inputValues.UseCustomTrueValue = filterArgs.value(use_custom_true_value); - return MultiThresholdObjects(dataStructure, messageHandler, shouldCancel, &inputValues)(); - -*/ - namespace nx::core { +/** + * @struct MultiThresholdObjectsInputValues + * @brief Input parameter bundle for the MultiThresholdObjects algorithm. + * + * Aggregates the threshold configuration, output mask settings, and custom + * true/false values needed by both the in-core (Direct) and out-of-core + * (Scanline) variants of multi-threshold filtering. + */ struct SIMPLNXCORE_EXPORT MultiThresholdObjectsInputValues { - ArrayThresholdsParameter::ValueType ArrayThresholdsObject; - DataTypeParameter::ValueType CreatedMaskType; - Float64Parameter::ValueType CustomFalseValue; - Float64Parameter::ValueType CustomTrueValue; - DataObjectNameParameter::ValueType OutputDataArrayName; - BoolParameter::ValueType UseCustomFalseValue; - BoolParameter::ValueType UseCustomTrueValue; + ArrayThresholdsParameter::ValueType ArrayThresholdsObject; ///< Tree of threshold comparisons (sets and individual conditions) + DataTypeParameter::ValueType CreatedMaskType; ///< DataType for the output mask array (e.g., uint8, bool) + Float64Parameter::ValueType CustomFalseValue; ///< Custom value to write for FALSE elements (default: 0.0) + Float64Parameter::ValueType CustomTrueValue; ///< Custom value to write for TRUE elements (default: 1.0) + DataObjectNameParameter::ValueType OutputDataArrayName; ///< Name of the output mask array + BoolParameter::ValueType UseCustomFalseValue; ///< Whether to use CustomFalseValue instead of 0 + BoolParameter::ValueType UseCustomTrueValue; ///< Whether to use CustomTrueValue instead of 1 }; /** * @class MultiThresholdObjects - * @brief This algorithm implements support code for the MultiThresholdObjectsFilter + * @brief Dispatcher algorithm for applying multiple threshold conditions to arrays + * and producing a boolean mask. + * + * This class acts as a thin dispatcher that selects between two concrete algorithm + * implementations at runtime: + * + * - **MultiThresholdObjectsDirect** (in-core): Uses per-element operator[] / getComponentValue() + * access with an O(n) tempResultVector for intermediate results. Optimal when all + * arrays reside in memory. + * + * - **MultiThresholdObjectsScanline** (out-of-core / OOC): Processes data in fixed-size + * 64K-tuple chunks using copyIntoBuffer()/copyFromBuffer() bulk I/O. Eliminates the + * O(n) tempResultVector by working chunk-by-chunk, and avoids per-element OOC access. + * + * The dispatch decision is made by DispatchAlgorithm() in + * AlgorithmDispatch.hpp, which checks whether any input IDataArray uses OOC storage. + * + * **Why two variants exist**: Each threshold condition requires reading an input array + * and comparing every element. When input arrays are stored out-of-core, per-element + * getComponentValue() calls trigger chunk load/evict cycles. The Scanline variant reads + * input data in 64K-tuple chunks, performs all comparisons for that chunk in memory, + * then writes the results back — converting N random accesses per threshold into + * N/65536 sequential bulk reads. + * + * @see MultiThresholdObjectsDirect + * @see MultiThresholdObjectsScanline + * @see AlgorithmDispatch.hpp */ - class SIMPLNXCORE_EXPORT MultiThresholdObjects { public: + /** + * @brief Constructs the dispatcher with all resources needed by either algorithm variant. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ MultiThresholdObjects(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, MultiThresholdObjectsInputValues* inputValues); ~MultiThresholdObjects() noexcept; @@ -55,13 +81,17 @@ class SIMPLNXCORE_EXPORT MultiThresholdObjects MultiThresholdObjects& operator=(const MultiThresholdObjects&) = delete; MultiThresholdObjects& operator=(MultiThresholdObjects&&) noexcept = delete; + /** + * @brief Dispatches to the Direct or Scanline algorithm based on storage type. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const MultiThresholdObjectsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const MultiThresholdObjectsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp index 3ba076ba8c..7dee5a2afa 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.cpp @@ -11,8 +11,38 @@ using namespace nx::core; +// ============================================================================= +// MultiThresholdObjectsDirect — In-Core Algorithm +// +// This file implements the in-core (Direct) variant of MultiThresholdObjects. +// It is selected by DispatchAlgorithm when all input arrays reside in memory. +// +// ALGORITHM OVERVIEW: +// For each threshold condition in the user-defined threshold tree: +// 1. Allocate an O(n) temporary result vector initialized to FALSE +// 2. Read each element of the input array via getComponentValue() +// 3. Apply the comparison (< > == !=) to produce TRUE/FALSE per element +// 4. Merge the temporary results into the output mask using AND/OR logic +// +// Threshold conditions can be nested in ArrayThresholdSets (which recursively +// apply AND/OR between their children) or be individual ArrayThreshold comparisons. +// +// DATA ACCESS PATTERN: +// Uses getComponentValue() for per-element random access to input arrays, and +// operator[] for per-element writes to the output mask and temporary vectors. +// This is optimal for in-memory data. The O(n) temporary vector is acceptable +// when data is in memory but would be wasteful for OOC data — see the Scanline +// variant which uses O(chunkSize) temporaries instead. +// ============================================================================= + namespace { +/** + * @brief Helper class that applies a single threshold comparison to an input array + * and writes TRUE/FALSE results into an output vector. + * + * @tparam U The output mask element type (e.g., uint8, bool, float32) + */ template class ThresholdFilterHelper { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp index ac17007ac8..a9b2894ed3 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsDirect.hpp @@ -11,13 +11,30 @@ struct MultiThresholdObjectsInputValues; /** * @class MultiThresholdObjectsDirect - * @brief In-core algorithm for MultiThresholdObjects. Preserves the original per-element - * access pattern and O(n) tempResultVector allocation. Selected by DispatchAlgorithm - * when all input arrays are backed by in-memory DataStore. + * @brief In-core algorithm for multi-threshold filtering using direct per-element array access. + * + * For each threshold condition, reads the input array via getComponentValue() and writes + * TRUE/FALSE into a temporary O(n) result vector. After evaluating a condition, the + * temporary results are merged into the output mask array using AND/OR logic. + * + * This is optimal when all arrays reside in memory, where getComponentValue() and + * operator[] are essentially pointer dereferences. + * + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory DataStore. + * + * @see MultiThresholdObjectsScanline for the out-of-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT MultiThresholdObjectsDirect { public: + /** + * @brief Constructs the in-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ MultiThresholdObjectsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const MultiThresholdObjectsInputValues* inputValues); ~MultiThresholdObjectsDirect() noexcept; @@ -26,13 +43,17 @@ class SIMPLNXCORE_EXPORT MultiThresholdObjectsDirect MultiThresholdObjectsDirect& operator=(const MultiThresholdObjectsDirect&) = delete; MultiThresholdObjectsDirect& operator=(MultiThresholdObjectsDirect&&) noexcept = delete; + /** + * @brief Executes the in-core multi-threshold filtering. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const MultiThresholdObjectsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const MultiThresholdObjectsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp index ee6da40666..42f3723cbd 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.cpp @@ -14,8 +14,45 @@ using namespace nx::core; +// ============================================================================= +// MultiThresholdObjectsScanline — Out-of-Core (OOC) Algorithm +// +// This file implements the out-of-core (Scanline) variant of MultiThresholdObjects. +// It is selected by DispatchAlgorithm when any input array uses chunked on-disk +// storage (e.g., ZarrStore / HDF5 chunked store). +// +// PROBLEM: +// The Direct variant uses getComponentValue() for per-element input reads and +// operator[] for per-element output writes, plus allocates an O(n) temporary +// result vector per threshold condition. When data is stored out-of-core: +// - Each getComponentValue() call may load an entire chunk from disk +// - The O(n) temporary vector wastes memory when only a chunk is needed +// - operator[] writes to the output mask may also trigger chunk load/evict +// +// SOLUTION — CHUNKED PROCESSING: +// Process data in fixed-size 64K-tuple chunks: +// 1. Read a chunk of the input array via copyIntoBuffer() (one bulk read) +// 2. Apply the threshold comparison to produce a chunk-sized temp buffer +// 3. For the first condition: write temp buffer to output via copyFromBuffer() +// 4. For subsequent conditions: read current output chunk, merge AND/OR, write back +// +// MEMORY SAVINGS: +// Peak memory per threshold condition is O(k_ChunkSize) = O(64K) instead of O(n). +// For a 100M-tuple dataset, this reduces temporary memory from ~100 MB to ~64 KB. +// +// IMPLEMENTATION NOTE: +// Temporary buffers use std::unique_ptr instead of std::vector to avoid +// the std::vector specialization, which would prevent direct memory access +// needed for copyIntoBuffer/copyFromBuffer spans. +// ============================================================================= + namespace { +/** + * @brief Chunk size for OOC processing. Each iteration reads/writes this many + * tuples via bulk I/O. 64K tuples balances between minimizing I/O calls and + * keeping per-chunk memory small. + */ constexpr usize k_ChunkSize = 65536; /** @@ -114,7 +151,30 @@ struct ChunkedThresholdHelper }; /** - * @brief Processes a single ArrayThreshold in chunks for OOC. + * @brief Processes a single ArrayThreshold comparison in chunks for OOC. + * + * For each 64K-tuple chunk: + * 1. Allocate a chunk-sized temp buffer (O(64K), not O(n)) + * 2. Read a chunk of the input array via ChunkedThresholdHelper (copyIntoBuffer) + * 3. Apply the comparison operator to fill the temp buffer with TRUE/FALSE + * 4. If this is the first condition (replaceInput=true): write the temp buffer + * directly to the output mask store via copyFromBuffer + * 5. If this is a subsequent condition: read the current output chunk via + * copyIntoBuffer, merge using AND/OR logic, then write back + * + * This chunk-by-chunk approach replaces the Direct variant's O(n) tempResultVector + * with an O(chunkSize) buffer, and replaces per-element input reads with bulk I/O. + * + * @tparam MaskT The output mask element type + * @param comparisonValue The threshold condition to evaluate + * @param dataStructure DataStructure containing the input array + * @param outputResultStore The output mask data store + * @param err Error code (set on failure) + * @param replaceInput If true, overwrite output; if false, merge with existing + * @param inverse If true, invert the comparison result before merging + * @param trueValue Value to write for TRUE elements + * @param falseValue Value to write for FALSE elements + * @param shouldCancel Cancellation flag */ template void ThresholdValueChunked(const ArrayThreshold& comparisonValue, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32& err, bool replaceInput, bool inverse, @@ -184,7 +244,14 @@ struct ThresholdValueChunkedFunctor }; /** - * @brief Processes an ArrayThresholdSet in chunks for OOC. + * @brief Processes an ArrayThresholdSet (a group of thresholds with AND/OR logic) in chunks for OOC. + * + * Recursively evaluates each child threshold in the set. Each child may be either + * a single ArrayThreshold (handled by ThresholdValueChunked) or a nested + * ArrayThresholdSet (handled recursively). The first child replaces the output; + * subsequent children merge using their AND/OR union operator. + * + * @tparam MaskT The output mask element type */ template void ThresholdSetChunked(const ArrayThresholdSet& inputComparisonSet, const DataStructure& dataStructure, AbstractDataStore& outputResultStore, int32& err, bool replaceInput, bool inverse, diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp index 7f6837ae10..09de2d70b8 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/MultiThresholdObjectsScanline.hpp @@ -11,14 +11,41 @@ struct MultiThresholdObjectsInputValues; /** * @class MultiThresholdObjectsScanline - * @brief Out-of-core algorithm for MultiThresholdObjects. Processes data in fixed-size - * chunks using copyIntoBuffer/copyFromBuffer bulk I/O to avoid per-element OOC access - * and eliminates the O(n) tempResultVector allocation. Selected by DispatchAlgorithm - * when any input array is backed by ZarrStore (out-of-core storage). + * @brief Out-of-core algorithm for multi-threshold filtering using chunked bulk I/O. + * + * Instead of the Direct variant's per-element access and O(n) temporary result vector, + * this variant processes data in fixed-size 64K-tuple chunks: + * + * For each threshold condition and each chunk: + * 1. Read a chunk of the input array via copyIntoBuffer() (sequential bulk read) + * 2. Apply the comparison operator to produce a chunk-sized temporary result buffer + * 3. For the first condition: write the temp buffer directly to the output mask + * via copyFromBuffer() + * 4. For subsequent conditions: read the current output chunk via copyIntoBuffer(), + * merge using AND/OR logic, then write back via copyFromBuffer() + * + * This approach has two advantages for OOC data: + * - All input array reads use sequential bulk I/O instead of per-element access + * - Peak memory per threshold condition is O(chunkSize) instead of O(n) + * + * The temporary buffers use std::unique_ptr instead of std::vector to avoid + * the std::vector specialization that would prevent direct memory access. + * + * Selected by DispatchAlgorithm when any input array is backed by out-of-core storage. + * + * @see MultiThresholdObjectsDirect for the in-core-optimized alternative. + * @see AlgorithmDispatch.hpp for the dispatch mechanism that selects between them. */ class SIMPLNXCORE_EXPORT MultiThresholdObjectsScanline { public: + /** + * @brief Constructs the out-of-core algorithm with all resources it needs. + * @param dataStructure The DataStructure containing input/output arrays + * @param mesgHandler Message handler for progress reporting + * @param shouldCancel Atomic flag checked periodically to support user cancellation + * @param inputValues Non-owning pointer to the parameter bundle + */ MultiThresholdObjectsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const MultiThresholdObjectsInputValues* inputValues); ~MultiThresholdObjectsScanline() noexcept; @@ -27,13 +54,17 @@ class SIMPLNXCORE_EXPORT MultiThresholdObjectsScanline MultiThresholdObjectsScanline& operator=(const MultiThresholdObjectsScanline&) = delete; MultiThresholdObjectsScanline& operator=(MultiThresholdObjectsScanline&&) noexcept = delete; + /** + * @brief Executes the OOC-optimized multi-threshold filtering. + * @return Result<> with any errors encountered during execution + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const MultiThresholdObjectsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure containing all arrays + const MultiThresholdObjectsInputValues* m_InputValues = nullptr; ///< Non-owning pointer to input parameters + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress updates }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp index fd79f774c1..6562d915e0 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.cpp @@ -1,3 +1,17 @@ +/** + * @file QuickSurfaceMesh.cpp + * @brief Dispatcher implementation for the QuickSurfaceMesh algorithm. + * + * This file contains the thin dispatch layer that examines the backing + * storage of the FeatureIds array and forwards execution to either: + * - QuickSurfaceMeshDirect -- when all arrays are in-memory (operator[]) + * - QuickSurfaceMeshScanline -- when any array uses chunked OOC storage + * + * The dispatch decision is made by DispatchAlgorithm, which inspects whether + * the DataStore is a chunked format. This avoids the 100-1000x performance + * penalty of random element access through virtual operator[] on OOC stores. + */ + #include "QuickSurfaceMesh.hpp" #include "QuickSurfaceMeshDirect.hpp" #include "QuickSurfaceMeshScanline.hpp" @@ -20,6 +34,15 @@ QuickSurfaceMesh::QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMes QuickSurfaceMesh::~QuickSurfaceMesh() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to the correct algorithm variant based on DataStore type. + * + * The FeatureIds array is the only input array whose access pattern matters + * for performance: the algorithm reads every voxel and its +X, +Y, +Z + * neighbors, which is sequential in Z-slice order but random across chunks + * when using OOC storage. All other arrays (triangle connectivity, vertex + * coordinates, face labels) are output-only and written sequentially. + */ Result<> QuickSurfaceMesh::operator()() { auto* featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp index a906eb20ba..f47ed5f73b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMesh.hpp @@ -13,28 +13,45 @@ namespace nx::core { +/** + * @struct QuickSurfaceMeshInputValues + * @brief Aggregates every user-facing parameter and internally-created + * DataPath needed by the QuickSurfaceMesh algorithm family. + */ struct SIMPLNXCORE_EXPORT QuickSurfaceMeshInputValues { - bool FixProblemVoxels; - bool RepairTriangleWinding; - bool GenerateTripleLines; - - DataPath GridGeomDataPath; - DataPath FeatureIdsArrayPath; - MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; - MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; - DataPath TriangleGeometryPath; - DataPath VertexGroupDataPath; - DataPath NodeTypesDataPath; - DataPath FaceGroupDataPath; - DataPath FaceLabelsDataPath; - MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; + bool FixProblemVoxels; ///< When true, run iterative problem-voxel correction before meshing + bool RepairTriangleWinding; ///< When true, run winding repair on the output triangle mesh + bool GenerateTripleLines; ///< When true, generate an EdgeGeom of triple lines (currently unused) + + DataPath GridGeomDataPath; ///< Path to the input IGridGeometry (ImageGeom or RectGridGeom) + DataPath FeatureIdsArrayPath; ///< Path to the Int32 FeatureIds cell array + MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; ///< Cell arrays to transfer to the triangle face attribute matrix + MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; ///< Feature arrays to transfer to the triangle face attribute matrix + DataPath TriangleGeometryPath; ///< Path to the created TriangleGeom output + DataPath VertexGroupDataPath; ///< Path to the vertex attribute matrix + DataPath NodeTypesDataPath; ///< Path to the Int8 NodeTypes vertex array + DataPath FaceGroupDataPath; ///< Path to the face attribute matrix + DataPath FaceLabelsDataPath; ///< Path to the Int32 FaceLabels (2-component) face array + MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; ///< Paths to the created face arrays (transferred cell/feature data) }; /** * @class QuickSurfaceMesh * @brief Dispatcher that selects between QuickSurfaceMeshDirect (in-core) and * QuickSurfaceMeshScanline (OOC) based on the storage type of input arrays. + * + * The algorithm generates a triangle mesh representing feature boundaries in + * a grid geometry. For every voxel face shared by two cells with different + * FeatureId values, two triangles are emitted. Boundary faces at the volume + * edges also produce triangles with FaceLabel = -1 on the exterior side. + * + * Dispatch is performed by DispatchAlgorithm: if the FeatureIds array is + * backed by an in-memory DataStore, QuickSurfaceMeshDirect is used. If it + * is backed by a chunked out-of-core store, QuickSurfaceMeshScanline is + * selected instead to avoid chunk thrashing. + * + * @see QuickSurfaceMeshDirect, QuickSurfaceMeshScanline */ class SIMPLNXCORE_EXPORT QuickSurfaceMesh { @@ -43,6 +60,13 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMesh using TriStore = AbstractDataStore; using MeshIndexType = IGeometry::MeshIndexType; + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all input/output objects + * @param inputValues Pointer to the parameter struct (must outlive this object) + * @param shouldCancel Atomic flag checked periodically for user cancellation + * @param mesgHandler Callback for progress and status messages + */ QuickSurfaceMesh(DataStructure& dataStructure, QuickSurfaceMeshInputValues* inputValues, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& mesgHandler); ~QuickSurfaceMesh() noexcept; @@ -51,12 +75,16 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMesh QuickSurfaceMesh& operator=(const QuickSurfaceMesh&) = delete; QuickSurfaceMesh& operator=(QuickSurfaceMesh&&) noexcept = delete; + /** + * @brief Dispatches to the appropriate in-core or OOC algorithm implementation. + * @return Result<> indicating success or an error code from the selected algorithm + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const QuickSurfaceMeshInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the active DataStructure + const QuickSurfaceMeshInputValues* m_InputValues = nullptr; ///< User parameters and created array paths + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message callback }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp index a3bc17a521..4c84c4309b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cpp @@ -1,3 +1,30 @@ +/** + * @file QuickSurfaceMeshDirect.cpp + * @brief In-core implementation of the QuickSurfaceMesh algorithm. + * + * This file contains the original QuickSurfaceMesh algorithm that uses direct + * operator[] access on DataStore references. It serves as the reference + * implementation for correctness. The algorithm generates a triangle surface + * mesh from a grid geometry by examining every voxel face: boundary faces + * (at volume edges) and interior faces where the FeatureId changes produce + * two triangles each. + * + * The algorithm proceeds in three major phases: + * Phase 1 (correctProblemVoxels): Fix diagonal-conflict voxel configurations + * Phase 2 (determineActiveNodes): Count nodes and triangles in one pass + * Phase 3 (createNodesAndTriangles): Write mesh data in a second pass + * + * For each voxel at grid position (i,j,k), the algorithm checks three neighbors: + * - neigh1 = (i+1, j, k) -- the +X neighbor + * - neigh2 = (i, j+1, k) -- the +Y neighbor + * - neigh3 = (i, j, k+1) -- the +Z neighbor + * + * If the FeatureIds differ across a face, or if the face is on the volume + * boundary, two triangles are generated for that face. The four vertices of + * the face are the corners of the dual grid (offset by +0.5 in each dimension + * from the cell centers). + */ + #include "QuickSurfaceMeshDirect.hpp" #include "QuickSurfaceMesh.hpp" @@ -19,6 +46,8 @@ using namespace nx::core; // ----------------------------------------------------------------------------- namespace { +// RNG constants for problem-voxel correction. The fixed seed ensures reproducible +// results across runs -- the same diagonal conflicts are resolved the same way. constexpr float64 k_RangeMin = 0.0; constexpr float64 k_RangeMax = 1.0; constexpr std::mt19937_64::result_type k_Seed = 3412341234123412; @@ -26,6 +55,11 @@ std::mt19937_64 generator(k_Seed); std::uniform_real_distribution<> distribution(k_RangeMin, k_RangeMax); // ----------------------------------------------------------------------------- +/** + * @brief Writes the world-space coordinates for a dual-grid vertex into the + * vertex coordinate array. The dual-grid vertex at integer (x,y,z) corresponds + * to the corner shared by up to 8 voxels in the primal grid. + */ void GetGridCoordinates(const IGridGeometry* grid, usize x, usize y, usize z, QuickSurfaceMeshDirect::VertexStore& verts, IGeometry::MeshIndexType nodeIndex) { nx::core::Point3D tmpCoords = grid->getPlaneCoords(x, y, z); @@ -35,6 +69,12 @@ void GetGridCoordinates(const IGridGeometry* grid, usize x, usize y, usize z, Qu } // ----------------------------------------------------------------------------- +/** + * @brief Resolves a Case 1 diagonal conflict: two voxels (v1 and v6) share + * a body diagonal but none of the 6 face-adjacent voxels match either. + * Randomly reassigns one of the two offending voxels to a face neighbor's value. + * The four outcomes have equal 25% probability each. + */ void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3, QuickSurfaceMeshDirect::MeshIndexType v4, QuickSurfaceMeshDirect::MeshIndexType v5, QuickSurfaceMeshDirect::MeshIndexType v6) { @@ -59,6 +99,12 @@ void FlipProblemVoxelCase1(Int32AbstractDataStore& featureIds, QuickSurfaceMeshD } // ----------------------------------------------------------------------------- +/** + * @brief Resolves a Case 2 edge-diagonal conflict: two voxels (v1 and v4) + * share an edge diagonal but neither of the two face-adjacent voxels on that + * edge match. Randomly reassigns one of the four voxels to break the diagonal + * with 8 equally-weighted outcomes (12.5% each). + */ void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3, QuickSurfaceMeshDirect::MeshIndexType v4) { @@ -99,6 +145,11 @@ void FlipProblemVoxelCase2(Int32AbstractDataStore& featureIds, QuickSurfaceMeshD } // ----------------------------------------------------------------------------- +/** + * @brief Resolves a Case 3 isolated-voxel conflict: one voxel (v1) is the only + * one that differs from all 7 other voxels in the 2x2x2 block. + * Randomly reassigns v2 or v3 to match v1 with 50/50 probability. + */ void FlipProblemVoxelCase3(Int32AbstractDataStore& featureIds, QuickSurfaceMeshDirect::MeshIndexType v1, QuickSurfaceMeshDirect::MeshIndexType v2, QuickSurfaceMeshDirect::MeshIndexType v3) { auto val = static_cast(distribution(generator)); @@ -129,6 +180,14 @@ QuickSurfaceMeshDirect::QuickSurfaceMeshDirect(DataStructure& dataStructure, con QuickSurfaceMeshDirect::~QuickSurfaceMeshDirect() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Executes the full in-core meshing pipeline. + * + * The algorithm allocates a nodeIds array of size (xP+1)*(yP+1)*(zP+1) -- + * one entry per possible dual-grid vertex. This maps grid-corner linear + * indices to sequential vertex IDs. The array starts filled with max values + * to indicate "not yet assigned." + */ Result<> QuickSurfaceMeshDirect::operator()() { auto& grid = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); @@ -140,6 +199,9 @@ Result<> QuickSurfaceMeshDirect::operator()() usize yP = udims[1]; usize zP = udims[2]; + // The dual grid has (xP+1)*(yP+1)*(zP+1) possible vertices. Each entry is + // initialized to max to indicate "unused." Active entries get assigned + // sequential vertex IDs during the counting pass. usize possibleNumNodes = (xP + 1) * (yP + 1) * (zP + 1); std::vector nodeIds(possibleNumNodes, std::numeric_limits::max()); @@ -205,6 +267,23 @@ Result<> QuickSurfaceMeshDirect::operator()() } // ----------------------------------------------------------------------------- +/** + * @brief Iteratively fixes problem voxels that would produce non-manifold geometry. + * + * A "problem voxel" configuration occurs when two voxels in a 2x2x2 block share + * a body diagonal or edge diagonal without any face-adjacent voxel sharing the + * same FeatureId. This creates degenerate zero-area triangles in the mesh. The + * fix randomly reassigns one of the conflicting voxels to break the diagonal. + * + * The iteration continues until no more problem voxels are found or 20 iterations + * are reached. Each iteration scans all 2x2x2 blocks in the volume. The 8 voxels + * in each block are labeled v1-v8 with v1-v4 in plane (k-1) and v5-v8 in plane k: + * + * v1 = (i-1, j-1, k-1) v5 = (i-1, j-1, k) + * v2 = (i, j-1, k-1) v6 = (i, j-1, k) + * v3 = (i-1, j, k-1) v7 = (i-1, j, k) + * v4 = (i, j, k-1) v8 = (i, j, k) + */ void QuickSurfaceMeshDirect::correctProblemVoxels() { m_MessageHandler(IFilter::Message::Type::Info, "Correcting Problem Voxels"); @@ -218,12 +297,15 @@ void QuickSurfaceMeshDirect::correctProblemVoxels() MeshIndexType yP = udims[1]; MeshIndexType zP = udims[2]; + // Linear indices into the FeatureIds array for the 8 voxels of a 2x2x2 block MeshIndexType v1 = 0, v2 = 0, v3 = 0, v4 = 0; MeshIndexType v5 = 0, v6 = 0, v7 = 0, v8 = 0; + // FeatureId values for the 8 voxels int32 f1 = 0, f2 = 0, f3 = 0, f4 = 0; int32 f5 = 0, f6 = 0, f7 = 0, f8 = 0; + // Row and plane offsets for computing linear indices MeshIndexType row1 = 0, row2 = 0; MeshIndexType plane1 = 0, plane2 = 0; @@ -376,6 +458,23 @@ void QuickSurfaceMeshDirect::correctProblemVoxels() } // ----------------------------------------------------------------------------- +/** + * @brief First pass: counts unique mesh vertices (nodes) and triangles. + * + * For each voxel, the algorithm checks six face conditions: + * - i==0, i==xP-1: left/right volume boundary faces + * - j==0, j==yP-1: front/back volume boundary faces + * - k==0, k==zP-1: bottom/top volume boundary faces + * - FeatureId differs from +X, +Y, or +Z neighbor: interior feature boundary + * + * Each face that produces triangles has 4 dual-grid nodes (corners of the + * face rectangle). The nodeIds array maps each possible dual-grid node + * (indexed by its (xP+1)*(yP+1)*(zP+1) linear position) to a sequential + * vertex ID. Unvisited entries contain max (sentinel value). + * + * Two triangles are generated per boundary face, so triangleCount is + * incremented by 2 for each detected face. + */ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& nodeIds, MeshIndexType& nodeCount, MeshIndexType& triangleCount) { m_MessageHandler(IFilter::Message::Type::Info, "Determining active Nodes"); @@ -389,8 +488,10 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no MeshIndexType yP = udims[1]; MeshIndexType zP = udims[2]; + // Linear indices: point = current voxel, neigh1/2/3 = +X, +Y, +Z neighbors MeshIndexType point = 0, neigh1 = 0, neigh2 = 0, neigh3 = 0; + // The 4 dual-grid node indices for the face being examined MeshIndexType nodeId1 = 0, nodeId2 = 0, nodeId3 = 0, nodeId4 = 0; for(MeshIndexType k = 0; k < zP; k++) @@ -675,6 +776,22 @@ void QuickSurfaceMeshDirect::determineActiveNodes(std::vector& no } // ----------------------------------------------------------------------------- +/** + * @brief Second pass: writes vertex coordinates, triangle connectivity, + * face labels, node types, and transferred cell/feature data. + * + * This pass mirrors the structure of determineActiveNodes but additionally: + * - Writes vertex world-space coordinates via GetGridCoordinates() + * - Writes triangle connectivity (3 vertex IDs per triangle) + * - Writes face labels (2-component: [lowerFeatureId, higherFeatureId]) + * with -1 for boundary faces + * - Tracks which features "own" each vertex in ownerLists for node type + * classification (2=interior, 3=triple line, 4=quad point, +10=boundary) + * - Runs TupleTransfer functions to copy cell/feature data to face arrays + * + * Face label ordering: the smaller FeatureId is always placed in component[0]. + * When one side is the volume exterior, FaceLabel[0] = -1. + */ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& m_NodeIds, MeshIndexType nodeCount, MeshIndexType triangleCount) { if(m_ShouldCancel) @@ -685,6 +802,7 @@ void QuickSurfaceMeshDirect::createNodesAndTriangles(std::vector& auto& featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); + // Scan all voxels to find the maximum FeatureId (needed for feature array sizing) usize numFeatures = 0; usize numTuples = featureIds.getNumberOfTuples(); for(usize i = 0; i < numTuples; i++) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp index f9e2388d30..02acce84de 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.hpp @@ -13,10 +13,29 @@ struct QuickSurfaceMeshInputValues; /** * @class QuickSurfaceMeshDirect - * @brief In-core algorithm for QuickSurfaceMesh. Preserves the original - * sequential voxel iteration using operator[] on DataStore references. - * Selected by DispatchAlgorithm when all input arrays are backed by - * in-memory DataStore. + * @brief In-core algorithm for QuickSurfaceMesh that uses direct operator[] + * access on DataStore references. + * + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory + * DataStore. This is the original algorithm implementation and serves as the + * reference for correctness. + * + * The algorithm runs in three phases: + * 1. **correctProblemVoxels** -- Iteratively resolves diagonal voxel + * configurations that would produce non-manifold mesh geometry by + * randomly reassigning one of the conflicting voxels to a neighbor's + * FeatureId. Up to 20 iterations until no problem voxels remain. + * 2. **determineActiveNodes** -- First pass over all voxels to count how + * many unique mesh vertices (nodes) and triangles will be needed. + * Allocates a nodeIds array of size (xP+1)*(yP+1)*(zP+1) to map + * grid-corner indices to sequential vertex IDs. + * 3. **createNodesAndTriangles** -- Second pass that writes vertex + * coordinates, triangle connectivity, face labels, node types, and + * transferred cell/feature data into the output TriangleGeom. + * + * Memory: O((xP+1)*(yP+1)*(zP+1)) for the nodeIds mapping array. + * + * @see QuickSurfaceMeshScanline for the OOC-optimized variant */ class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect { @@ -25,6 +44,13 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect using TriStore = AbstractDataStore; using MeshIndexType = IGeometry::MeshIndexType; + /** + * @brief Constructs the in-core algorithm. Seeds the RNG used by problem-voxel correction. + * @param dataStructure The DataStructure containing all input/output objects + * @param mesgHandler Callback for progress and status messages + * @param shouldCancel Atomic flag checked periodically for user cancellation + * @param inputValues Pointer to the parameter struct (must outlive this object) + */ QuickSurfaceMeshDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const QuickSurfaceMeshInputValues* inputValues); ~QuickSurfaceMeshDirect() noexcept; @@ -33,17 +59,45 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMeshDirect QuickSurfaceMeshDirect& operator=(const QuickSurfaceMeshDirect&) = delete; QuickSurfaceMeshDirect& operator=(QuickSurfaceMeshDirect&&) noexcept = delete; + /** + * @brief Executes the full in-core meshing pipeline: problem voxel correction, + * node counting, mesh generation, and optional winding repair. + * @return Result<> indicating success or an error from winding repair + */ Result<> operator()(); private: + /** + * @brief Iteratively resolves non-manifold diagonal voxel configurations. + * + * Examines every 2x2x2 block of voxels and detects configurations where + * diagonally-opposite voxels share a FeatureId but no face-adjacent voxel + * does. These produce degenerate triangles. The fix randomly reassigns one + * voxel to a neighbor's value using a seeded RNG for reproducibility. + */ void correctProblemVoxels(); + + /** + * @brief First pass: counts active mesh nodes and triangles. + * @param[in,out] nodeIds Maps grid-corner linear index to sequential vertex ID (initially max) + * @param[out] nodeCount Total number of unique mesh vertices found + * @param[out] triangleCount Total number of triangles that will be generated + */ void determineActiveNodes(std::vector& nodeIds, MeshIndexType& nodeCount, MeshIndexType& triangleCount); + + /** + * @brief Second pass: writes vertex coordinates, triangle connectivity, + * face labels, node types, and runs TupleTransfer for cell/feature arrays. + * @param[in] nodeIds Grid-corner-to-vertex mapping from determineActiveNodes + * @param nodeCount Number of vertices (used for resizing) + * @param triangleCount Number of triangles (used for resizing) + */ void createNodesAndTriangles(std::vector& nodeIds, MeshIndexType nodeCount, MeshIndexType triangleCount); - DataStructure& m_DataStructure; - const QuickSurfaceMeshInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the active DataStructure + const QuickSurfaceMeshInputValues* m_InputValues = nullptr; ///< User parameters and created array paths + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message callback }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp index 4e0a829b40..732563b7ab 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.cpp @@ -1,3 +1,50 @@ +/** + * @file QuickSurfaceMeshScanline.cpp + * @brief Out-of-core (OOC) optimized implementation of the QuickSurfaceMesh algorithm. + * + * This file implements the scanline variant of QuickSurfaceMesh that avoids + * per-element random access to chunked DataStores. Instead of using operator[] + * on the FeatureIds array (which triggers chunk load/evict cycles on OOC stores), + * this algorithm reads exactly two Z-slices at a time via copyIntoBuffer() and + * processes them entirely in local buffers. + * + * ## Key Differences from QuickSurfaceMeshDirect + * + * 1. **FeatureIds access**: Direct uses operator[], Scanline uses copyIntoBuffer() + * to bulk-read one Z-slice (xP * yP elements) at a time. + * + * 2. **Node ID mapping**: Direct allocates an O(volume) nodeIds array of size + * (xP+1)*(yP+1)*(zP+1). Scanline uses two rolling node-plane buffers of + * size O((xP+1)*(yP+1)) each, swapped after each Z-slice. This reduces + * memory from O(volume) to O(slice). + * + * 3. **Output writes**: Direct writes vertices, triangles, and face labels + * per-element via operator[]. Scanline buffers all vertex coordinates in + * a single allocation and flushes once, while triangle connectivity and + * face labels are buffered per-slice and flushed via copyFromBuffer(). + * + * 4. **Problem voxel correction**: Direct reads/writes the DataStore directly. + * Scanline loads Z-slice pairs, mutates the local buffers, and only writes + * back slices that were actually modified (dirty flag optimization). + * + * 5. **TupleTransfer**: Direct calls quickSurfaceTransfer() per-triangle. + * Scanline uses quickSurfaceTransferBatch() to process all triangles from + * a Z-slice in one call, reducing virtual function call overhead. + * + * ## Rolling Buffer Diagram + * + * For Z-slice k, the algorithm needs: + * - curSlice[]: FeatureIds for Z = k (xP * yP elements) + * - nextSlice[]: FeatureIds for Z = k+1 (xP * yP elements) + * - nodePlane0[]: Vertex IDs for Z = k ((xP+1) * (yP+1) elements) + * - nodePlane1[]: Vertex IDs for Z = k+1 ((xP+1) * (yP+1) elements) + * + * After processing slice k: + * - std::swap(curSlice, nextSlice) -- old next becomes current + * - std::swap(nodePlane0, nodePlane1) -- plane1 becomes plane0 + * - nodePlane1 is reset to sentinel values + */ + #include "QuickSurfaceMeshScanline.hpp" #include "QuickSurfaceMesh.hpp" @@ -20,13 +67,17 @@ using namespace nx::core; // ----------------------------------------------------------------------------- namespace { +// RNG constants -- must match QuickSurfaceMeshDirect.cpp to produce identical +// problem-voxel corrections (same seed, same sequence of random draws). constexpr float64 k_RangeMin = 0.0; constexpr float64 k_RangeMax = 1.0; constexpr std::mt19937_64::result_type k_Seed = 3412341234123412; std::mt19937_64 generator(k_Seed); std::uniform_real_distribution<> distribution(k_RangeMin, k_RangeMax); -// Buffer-based flip functions that operate on raw int32 pointers +// Buffer-based flip functions that operate on raw int32 pointers into local +// slice buffers rather than through the DataStore. These are functionally +// identical to the Direct variant's flip functions but take raw pointers. // ----------------------------------------------------------------------------- void FlipProblemVoxelCase1(int32* buf, QuickSurfaceMeshScanline::MeshIndexType v1, QuickSurfaceMeshScanline::MeshIndexType v2, QuickSurfaceMeshScanline::MeshIndexType v3, QuickSurfaceMeshScanline::MeshIndexType v4, QuickSurfaceMeshScanline::MeshIndexType v5, QuickSurfaceMeshScanline::MeshIndexType v6) @@ -122,6 +173,17 @@ QuickSurfaceMeshScanline::QuickSurfaceMeshScanline(DataStructure& dataStructure, QuickSurfaceMeshScanline::~QuickSurfaceMeshScanline() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Executes the full OOC meshing pipeline. + * + * Orchestrates three phases: problem voxel correction, node/triangle counting, + * and mesh generation. Between the counting and generation phases, the output + * TriangleGeom arrays are resized to their final sizes. After mesh generation, + * optional winding repair is applied. + * + * All FeatureIds access uses copyIntoBuffer() for bulk Z-slice reads. + * All output writes use copyFromBuffer() for bulk sequential writes. + */ Result<> QuickSurfaceMeshScanline::operator()() { auto& grid = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); @@ -137,6 +199,7 @@ Result<> QuickSurfaceMeshScanline::operator()() MeshIndexType triangleCount = 0; usize numFeatures = 0; + // Phase 1: Fix diagonal-conflict voxel configurations (optional) if(m_InputValues->FixProblemVoxels) { correctProblemVoxels(); @@ -191,6 +254,27 @@ Result<> QuickSurfaceMeshScanline::operator()() } // ----------------------------------------------------------------------------- +/** + * @brief OOC problem-voxel correction using double-buffered Z-slice pairs. + * + * This is the OOC equivalent of QuickSurfaceMeshDirect::correctProblemVoxels(). + * Instead of using operator[] to read/write the FeatureIds DataStore directly, + * it loads two adjacent Z-slices (sliceA = k-1, sliceB = k) into local buffers + * via copyIntoBuffer(), performs all the same diagonal-conflict checks and + * random reassignments in local memory, then writes back only modified slices + * via copyFromBuffer() using dirty flags. + * + * The problem voxel checks examine 2x2x2 blocks where the 8 voxels span two + * adjacent Z-slices. voxels v1-v4 are in sliceA (Z = k-1) and v5-v8 are in + * sliceB (Z = k). The Case1/Case2/Case3 flip logic is inlined rather than + * delegated to helper functions because the mutations may target either + * sliceA or sliceB, and we need to track which slice was modified. + * + * The doCase2 lambda handles Case2 variants that may target 4 voxels across + * either or both slices. It takes buffer pointers and dirty-flag references + * for each of the 4 voxel positions, allowing it to work with any combination + * of sliceA and sliceB targets. + */ void QuickSurfaceMeshScanline::correctProblemVoxels() { m_MessageHandler(IFilter::Message::Type::Info, "Correcting Problem Voxels"); @@ -206,7 +290,8 @@ void QuickSurfaceMeshScanline::correctProblemVoxels() const MeshIndexType sliceSize = xP * yP; - // Buffer two consecutive z-slices at a time + // Double-buffered Z-slices: sliceA holds Z = (k-1), sliceB holds Z = k. + // Each buffer is xP * yP elements (one full Z-slice of FeatureIds). auto sliceA = std::make_unique(sliceSize); auto sliceB = std::make_unique(sliceSize); @@ -555,6 +640,33 @@ void QuickSurfaceMeshScanline::correctProblemVoxels() } // ----------------------------------------------------------------------------- +/** + * @brief Counting pass using rolling 2-plane node buffers and double-buffered + * FeatureId Z-slices. + * + * This is the OOC equivalent of QuickSurfaceMeshDirect::determineActiveNodes(). + * The key difference is memory reduction: instead of an O(volume) nodeIds array, + * this method uses two node-plane buffers of size O((xP+1)*(yP+1)) each. + * + * ## Rolling Buffer Strategy + * + * For Z-slice k, the dual-grid nodes lie on two planes: + * - nodePlane0: nodes at Z = k (the "current" plane) + * - nodePlane1: nodes at Z = k+1 (the "next" plane) + * + * After processing all voxels in slice k: + * 1. nodePlane0 is discarded (all its nodes have been assigned) + * 2. nodePlane1 becomes nodePlane0 for the next iteration + * 3. A fresh nodePlane1 is initialized with sentinel values + * + * This works because each node is referenced only by voxels at Z = k and Z = k-1. + * Once we advance past Z = k, nodes in the Z = k plane are never accessed again. + * + * The FeatureIds are double-buffered similarly: curSlice holds Z = k, nextSlice + * holds Z = k+1. After processing, they swap so the old next becomes current. + * + * Also tracks the maximum FeatureId value (numFeatures) for later array sizing. + */ void QuickSurfaceMeshScanline::countActiveNodesAndTriangles(MeshIndexType& nodeCount, MeshIndexType& triangleCount, usize& numFeatures) { m_MessageHandler(IFilter::Message::Type::Info, "Counting active nodes and triangles"); @@ -572,11 +684,14 @@ void QuickSurfaceMeshScanline::countActiveNodesAndTriangles(MeshIndexType& nodeC const MeshIndexType nodePlaneSize = (xP + 1) * (yP + 1); constexpr auto kMax = std::numeric_limits::max(); - // Rolling node-plane buffers: O(2 * nodePlaneSize) instead of O((xP+1)*(yP+1)*(zP+1)) + // Rolling node-plane buffers: O(2 * nodePlaneSize) instead of O((xP+1)*(yP+1)*(zP+1)). + // nodePlane0 corresponds to Z = k (current), nodePlane1 to Z = k+1 (next). + // Entries start at kMax (sentinel) and are assigned sequential IDs on first use. std::vector nodePlane0(nodePlaneSize, kMax); std::vector nodePlane1(nodePlaneSize, kMax); - // Lambda to count a node: if not yet assigned, assign and increment + // Lambda to count a node: if not yet assigned in this plane, assign the + // next sequential vertex ID and increment the counter. auto countNode = [&](std::vector& plane, MeshIndexType offset) { if(plane[offset] == kMax) { @@ -585,11 +700,11 @@ void QuickSurfaceMeshScanline::countActiveNodesAndTriangles(MeshIndexType& nodeC } }; - // Buffer current and next z-slices for featureIds + // Double-buffered FeatureId Z-slices: curSlice holds Z = k, nextSlice holds Z = k+1. auto curSlice = std::make_unique(sliceSize); auto nextSlice = std::make_unique(sliceSize); - // Load first slice + // Load the first Z-slice (k = 0) via bulk I/O featureIdsStore.copyIntoBuffer(0, nonstd::span(curSlice.get(), sliceSize)); numFeatures = 0; @@ -708,6 +823,40 @@ void QuickSurfaceMeshScanline::countActiveNodesAndTriangles(MeshIndexType& nodeC } // ----------------------------------------------------------------------------- +/** + * @brief Generation pass: creates the output triangle mesh with OOC-safe I/O. + * + * This is the OOC equivalent of QuickSurfaceMeshDirect::createNodesAndTriangles(). + * It produces identical output but uses bulk I/O for all reads and writes: + * + * ## Output Buffering Strategy + * + * - **Vertex coordinates**: Buffered in vertCoordBuf (nodeCount * 3 floats), + * flushed once at the end via a single copyFromBuffer() call. This avoids + * per-vertex OOC writes. The buffer is O(surface) not O(volume) because + * only boundary nodes get vertices. + * + * - **Triangle connectivity**: Accumulated in triBuffer per Z-slice, flushed + * via copyFromBuffer() at the end of each slice. This keeps peak memory + * proportional to the number of triangles in one Z-slice. + * + * - **Face labels**: Accumulated in faceLabelBuf per Z-slice, flushed with + * triangle connectivity. + * + * - **TupleTransfer**: Arguments accumulated in ttArgsBuf per Z-slice and + * flushed via quickSurfaceTransferBatch() to reduce virtual call overhead. + * + * - **Node types**: Computed from ownerLists after all triangles are generated, + * buffered in a local array, and flushed via one copyFromBuffer() call. + * + * ## Node Assignment + * + * Uses the same rolling 2-plane node buffer strategy as countActiveNodesAndTriangles(). + * The assignNode lambda both assigns sequential vertex IDs and writes vertex + * coordinates into vertCoordBuf. Multiple calls to assignNode with the same + * plane offset are harmless -- the coordinate write is idempotent (last-write-wins + * matches the Direct variant's behavior). + */ void QuickSurfaceMeshScanline::createNodesAndTriangles(MeshIndexType nodeCount, MeshIndexType triangleCount, usize numFeatures) { if(m_ShouldCancel) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp index 984ff950a6..072d1c2ff6 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshScanline.hpp @@ -15,15 +15,52 @@ struct QuickSurfaceMeshInputValues; /** * @class QuickSurfaceMeshScanline - * @brief Out-of-core algorithm for QuickSurfaceMesh. Selected by - * DispatchAlgorithm when any input array is backed by chunked (OOC) storage. - * - * Buffers featureIds in z-slice pairs using copyIntoBuffer to avoid - * per-element virtual dispatch through AbstractDataStore::operator[]. - * The correctProblemVoxels pass uses double-buffered z-slice pairs with - * copyFromBuffer write-back. The countActiveNodesAndTriangles and - * createNodesAndTriangles passes use rolling 2-plane node buffers of - * size O((xP+1)*(yP+1)) instead of the O(volume) nodeIds array. + * @brief Out-of-core (OOC) optimized algorithm for QuickSurfaceMesh. + * + * Selected by DispatchAlgorithm when any input array is backed by chunked + * (OOC) storage. Produces identical output to QuickSurfaceMeshDirect but + * avoids random-access element reads that would cause chunk thrashing on + * disk-backed DataStores. + * + * ## OOC Strategy + * + * The key insight is that the QuickSurfaceMesh algorithm only compares each + * voxel with its +X, +Y, and +Z neighbors. This means at most two adjacent + * Z-slices of FeatureIds are needed at any time. The scanline variant + * exploits this by: + * + * - **Bulk I/O**: Reading FeatureIds one Z-slice at a time via + * copyIntoBuffer() instead of per-element operator[]. This converts + * O(volume) random reads into O(zP) sequential bulk reads. + * + * - **Rolling node-plane buffers**: Instead of the O((xP+1)*(yP+1)*(zP+1)) + * nodeIds array used by the Direct variant, this algorithm maintains two + * node-plane buffers of size O((xP+1)*(yP+1)) that roll forward as each + * Z-slice is processed. This reduces memory from O(volume) to O(slice). + * + * - **Buffered output writes**: Triangle connectivity, face labels, and + * vertex coordinates are accumulated in per-slice buffers and flushed + * via copyFromBuffer() in one bulk write per Z-slice, avoiding + * per-element OOC writes. + * + * ## Phases + * + * 1. **correctProblemVoxels** -- Same diagonal-voxel fix as the Direct + * variant, but reads/writes Z-slice pairs via copyIntoBuffer/copyFromBuffer. + * Uses dirty flags to skip write-back for unmodified slices. + * + * 2. **countActiveNodesAndTriangles** -- Counting pass using rolling + * node-plane buffers and double-buffered FeatureId slices. + * + * 3. **createNodesAndTriangles** -- Generation pass that writes mesh data. + * Vertex coordinates are buffered in a single allocation of size + * O(nodeCount * 3) and flushed once at the end. Triangle connectivity + * and face labels are flushed per-slice. + * + * Memory: O((xP+1)*(yP+1)) for node planes + O(xP*yP) for FeatureId slices + * + O(nodeCount * 3) for vertex coordinate buffer. + * + * @see QuickSurfaceMeshDirect for the in-core reference implementation */ class SIMPLNXCORE_EXPORT QuickSurfaceMeshScanline { @@ -32,6 +69,13 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMeshScanline using TriStore = AbstractDataStore; using MeshIndexType = IGeometry::MeshIndexType; + /** + * @brief Constructs the OOC-optimized algorithm. Seeds the RNG for problem-voxel correction. + * @param dataStructure The DataStructure containing all input/output objects + * @param mesgHandler Callback for progress and status messages + * @param shouldCancel Atomic flag checked periodically for user cancellation + * @param inputValues Pointer to the parameter struct (must outlive this object) + */ QuickSurfaceMeshScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const QuickSurfaceMeshInputValues* inputValues); ~QuickSurfaceMeshScanline() noexcept; @@ -40,17 +84,51 @@ class SIMPLNXCORE_EXPORT QuickSurfaceMeshScanline QuickSurfaceMeshScanline& operator=(const QuickSurfaceMeshScanline&) = delete; QuickSurfaceMeshScanline& operator=(QuickSurfaceMeshScanline&&) noexcept = delete; + /** + * @brief Executes the full OOC meshing pipeline: problem voxel correction, + * node/triangle counting, mesh generation, and optional winding repair. + * @return Result<> indicating success or an error from winding repair + */ Result<> operator()(); private: + /** + * @brief OOC problem-voxel correction using double-buffered Z-slice pairs. + * + * Reads two adjacent Z-slices at a time via copyIntoBuffer(), applies the + * same 2x2x2 diagonal-conflict resolution as the Direct variant, then + * writes back only the slices that were actually modified (dirty flags). + * This avoids per-element read/write through the OOC DataStore. + */ void correctProblemVoxels(); + + /** + * @brief Counting pass: determines total node and triangle counts using + * rolling 2-plane node buffers and double-buffered FeatureId Z-slices. + * @param[out] nodeCount Total number of unique mesh vertices + * @param[out] triangleCount Total number of triangles to generate + * @param[out] numFeatures Maximum FeatureId value found (used for feature array sizing) + */ void countActiveNodesAndTriangles(MeshIndexType& nodeCount, MeshIndexType& triangleCount, usize& numFeatures); + + /** + * @brief Generation pass: creates vertices, triangles, face labels, node types, + * and runs TupleTransfer for cell/feature data arrays. + * + * Uses rolling node-plane buffers to assign vertex IDs, a single bulk + * vertex-coordinate buffer flushed at the end, and per-slice buffers for + * triangle connectivity and face labels flushed via copyFromBuffer(). + * + * @param nodeCount Number of vertices from counting pass (for buffer allocation) + * @param triangleCount Number of triangles from counting pass (for resizing) + * @param numFeatures Maximum FeatureId (for feature array sizing) + */ void createNodesAndTriangles(MeshIndexType nodeCount, MeshIndexType triangleCount, usize numFeatures); - DataStructure& m_DataStructure; - const QuickSurfaceMeshInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the active DataStructure + const QuickSurfaceMeshInputValues* m_InputValues = nullptr; ///< User parameters and created array paths + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message callback }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp index 9b7c9d7243..fd5f8693ac 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.cpp @@ -14,6 +14,9 @@ using namespace nx::core; namespace fs = std::filesystem; // ----------------------------------------------------------------------------- +/** + * @brief Constructs ReadHDF5Dataset with the given DataStructure, message handler, cancel flag, and input values. + */ ReadHDF5Dataset::ReadHDF5Dataset(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, ReadHDF5DatasetInputValues* inputValues) : m_DataStructure(dataStructure) , m_InputValues(inputValues) @@ -26,6 +29,21 @@ ReadHDF5Dataset::ReadHDF5Dataset(DataStructure& dataStructure, const IFilter::Me ReadHDF5Dataset::~ReadHDF5Dataset() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Reads one or more HDF5 datasets from the input file and populates the DataStructure. + * + * Each selected dataset is read via HDF5::Support::FillDataArray, which internally uses + * the HDF5 library for bulk reads. Progress messages report the current dataset index + * and name. Cancel checking occurs between datasets so that long multi-dataset imports + * can be interrupted. + * + * @section ooc_note OOC Note + * The OOC changes here are minor (progress messaging and cancel support between datasets). + * The actual HDF5 -> DataStore transfer is handled by the FillDataArray utility which + * already uses bulk I/O internally. + * + * @return Result<> indicating success or an error from the HDF5 reading infrastructure. + */ Result<> ReadHDF5Dataset::operator()() { auto pSelectedAttributeMatrixValue = m_InputValues->ImportHdf5Object.parent; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.hpp index 2f1a967c07..72d2c88d8f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadHDF5Dataset.hpp @@ -7,27 +7,25 @@ #include "simplnx/Filter/IFilter.hpp" #include "simplnx/Parameters/ReadHDF5DatasetParameter.hpp" -/** -* This is example code to put in the Execute Method of the filter. - ReadHDF5DatasetInputValues inputValues; - inputValues.ImportHdf5Object = filterArgs.value(import_hdf5_object); - return ReadHDF5Dataset(dataStructure, messageHandler, shouldCancel, &inputValues)(); - -*/ - namespace nx::core { +/** + * @brief Input values for the ReadHDF5Dataset algorithm. + */ struct SIMPLNXCORE_EXPORT ReadHDF5DatasetInputValues { - ReadHDF5DatasetParameter::ValueType ImportHdf5Object; + ReadHDF5DatasetParameter::ValueType ImportHdf5Object; ///< HDF5 import configuration (file path, parent, dataset list). }; /** * @class ReadHDF5Dataset - * @brief This algorithm implements support code for the ReadHDF5DatasetFilter + * @brief Reads one or more datasets from an HDF5 file into the DataStructure. + * + * Iterates over the user-selected datasets, reads each via the HDF5 support library, + * and populates the corresponding DataArray. Progress messages report the current + * dataset being imported. */ - class SIMPLNXCORE_EXPORT ReadHDF5Dataset { public: diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp index 6a96929290..0f20785548 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.cpp @@ -143,6 +143,10 @@ Result<> ReadStlFile::operator()() fpos_t pos; + // Progress reporting is throttled to every k_ProgressStride triangles to reduce + // the overhead of the modulo check and message formatting in the tight read loop. + // The original code called sendThrottledMessage on every iteration, which added + // measurable overhead when reading millions of triangles. constexpr int32_t k_ProgressStride = 10000; for(int32_t t = 0; t < triCount; ++t) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.hpp index 9b0c50aaf6..48f85306b7 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadStlFile.hpp @@ -15,8 +15,12 @@ namespace fs = std::filesystem; namespace nx::core { /** - * @class ConditionalSetValueFilter - + * @class ReadStlFile + * @brief Reads a binary STL mesh file into a TriangleGeom, creating vertices, faces, and face normals. + * + * Handles vendor-specific STL quirks (Magics color encoding, VxElements metadata) by + * detecting them from the header and adjusting the parse behavior accordingly. + * After reading, calls GeometryUtilities::EliminateDuplicateNodes to merge shared vertices. */ class SIMPLNXCORE_EXPORT ReadStlFile { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp index 51a7b9c61f..836c0a09ce 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.cpp @@ -1,3 +1,45 @@ +/** + * @file ReplaceElementAttributesWithNeighborValues.cpp + * @brief Threshold-based neighbor replacement algorithm, optimized for + * out-of-core (OOC) data stores via Z-slice buffered I/O. + * + * ## High-Level Flow (per pass) + * + * 1. **Initialize rolling window** -- Load input array Z-slices 0 and 1 + * into slots 1 (current) and 2 (next) of the three-element window. + * + * 2. **Scan every voxel** (Z-major, then Y, then X): + * - If the voxel's value fails the threshold comparison (e.g., confidence + * index < 0.1), examine its 6 face neighbors via the rolling window. + * - Among neighbors that pass the threshold, track the one with the + * best (most favorable) value. + * - Record that neighbor's global index in the per-slice mark array. + * + * 3. **Immediate per-slice transfer** -- After each Z-slice's XY scan + * completes, commit the marks via SliceBufferedTransferOneZ for every + * array in the Attribute Matrix. This is safe because the marks for + * this algorithm always point to face neighbors (within +/- 1 Z-slice), + * and the best-neighbor mark only writes to the current voxel (not + * across slices like dilation does). + * + * 4. **Repeat** -- If Loop is true and any voxels were modified, start a + * new pass. Re-read the rolling window because transfers changed values. + * + * ## Comparison Functors + * + * The algorithm supports two comparison modes via a polymorphic functor: + * - **LessThanComparison**: Targets voxels whose value < threshold. Prefers + * neighbors with the highest value (closest to passing). + * - **GreaterThanComparison**: Targets voxels whose value > threshold. Prefers + * neighbors with the lowest value. + * + * Each functor provides three comparison methods: + * - `compare(value, threshold)`: Does this voxel fail the threshold? + * - `compare1(neighborValue, threshold)`: Does this neighbor pass? + * - `compare2(neighborValue, bestSoFar)`: Is this neighbor better than the + * current best? + */ + #include "ReplaceElementAttributesWithNeighborValues.hpp" #include "simplnx/DataStructure/DataArray.hpp" @@ -12,6 +54,16 @@ namespace { const int32 k_GreaterThanIndex = 1; +/** + * @class IComparisonFunctor + * @brief Abstract base for threshold comparison strategies. + * @tparam T The element type of the input array being compared. + * + * Provides three virtual comparison methods that together define: + * - Whether a voxel fails the threshold (compare) + * - Whether a neighbor passes the threshold (compare1) + * - Whether a neighbor is a better replacement than the current best (compare2) + */ template class IComparisonFunctor { @@ -24,11 +76,23 @@ class IComparisonFunctor IComparisonFunctor& operator=(const IComparisonFunctor&) = delete; // Copy Assignment Not Implemented IComparisonFunctor& operator=(IComparisonFunctor&&) = delete; // Move Assignment Not Implemented + /** @brief Returns true if `left` fails the threshold relative to `right`. */ [[nodiscard]] virtual bool compare(T left, T right) const = 0; + /** @brief Returns true if `left` (a neighbor value) passes the threshold `right`. */ [[nodiscard]] virtual bool compare1(T left, T right) const = 0; + /** @brief Returns true if `left` is a better replacement candidate than `right`. */ [[nodiscard]] virtual bool compare2(T left, T right) const = 0; }; +/** + * @class LessThanComparison + * @brief Targets voxels below the threshold; prefers neighbors with higher values. + * @tparam T The element type of the input array. + * + * - compare: value < threshold (voxel is below cutoff) + * - compare1: neighbor >= threshold (neighbor passes) + * - compare2: neighbor > best (neighbor is closer to ideal) + */ template class LessThanComparison : public IComparisonFunctor { @@ -55,6 +119,15 @@ class LessThanComparison : public IComparisonFunctor } }; +/** + * @class GreaterThanComparison + * @brief Targets voxels above the threshold; prefers neighbors with lower values. + * @tparam T The element type of the input array. + * + * - compare: value > threshold (voxel is above cutoff) + * - compare1: neighbor <= threshold (neighbor passes) + * - compare2: neighbor < best (neighbor is closer to ideal) + */ template class GreaterThanComparison : public IComparisonFunctor { @@ -80,8 +153,28 @@ class GreaterThanComparison : public IComparisonFunctor } }; +/** + * @struct ExecuteTemplate + * @brief Type-dispatched functor that implements the core threshold-replacement + * algorithm with OOC-optimized Z-slice buffering. + * + * This struct is invoked via ExecuteDataFunction, which instantiates operator() + * with the correct element type T matching the input array's DataType. + */ struct ExecuteTemplate { + /** + * @brief Checks whether a neighbor value qualifies as a replacement and + * updates the best-neighbor tracking state if so. + * @tparam T Element type of the input array + * @param comparator The active comparison functor + * @param neighborValue The neighbor's value from the rolling-window buffer + * @param ThresholdValue The user's threshold cutoff + * @param best [in/out] The best candidate value found so far + * @param bestNeighbor [in/out] Per-slice mark array tracking the best neighbor + * @param i In-slice index of the current voxel + * @param neighborPoint Global flat index of the neighbor voxel + */ template void CompareValues(std::shared_ptr>& comparator, T neighborValue, float32 ThresholdValue, float32& best, std::vector& bestNeighbor, usize i, int64 neighborPoint) const { @@ -92,6 +185,18 @@ struct ExecuteTemplate } } + /** + * @brief Core algorithm: iteratively replaces voxels that fail a threshold + * comparison with the best-scoring face neighbor's data. + * @tparam T Element type of the input array + * @param imageGeom The ImageGeom defining the voxel grid dimensions + * @param inputIDataArray Pointer to the input data array (used for comparison) + * @param comparisonAlgorithm 0 = LessThan, 1 = GreaterThan + * @param ThresholdValue The user's threshold cutoff value + * @param loopUntilDone If true, repeat passes until no failing voxels remain + * @param shouldCancel Atomic cancellation flag + * @param messageHandler Handler for progress messages + */ template void operator()(const ImageGeom& imageGeom, IDataArray* inputIDataArray, int32 comparisonAlgorithm, float32 ThresholdValue, bool loopUntilDone, const std::atomic_bool& shouldCancel, const IFilter::MessageHandler& messageHandler) @@ -107,30 +212,37 @@ struct ExecuteTemplate static_cast(udims[2]), }; + // Precompute face-neighbor index offsets and iteration order std::array neighborVoxelIndexOffsets = initializeFaceNeighborOffsets(dims); std::array faceNeighborInternalIdx = initializeFaceNeighborInternalIdx(); usize count = 0; bool keepGoing = true; + // Select the comparison strategy based on user choice std::shared_ptr> comparator = std::make_shared>(); if(comparisonAlgorithm == k_GreaterThanIndex) { comparator = std::make_shared>(); } + // The Attribute Matrix holds all sibling arrays that should be updated + // together when a voxel is replaced (e.g., orientations, phases, etc.) const AttributeMatrix* attrMatrix = imageGeom.getCellData(); - // Z-slice buffering: maintain rolling window of 3 adjacent Z-slices for input array - // to avoid random OOC chunk access during neighbor lookups. + // ---- Z-slice buffering setup ---- const usize sliceSize = static_cast(dims[0]) * static_cast(dims[1]); const usize dimZ = static_cast(dims[2]); - // Per-slice best neighbor marks (replaces O(totalPoints) bestNeighbor array) + // Per-slice best-neighbor marks: O(sliceSize) instead of O(totalPoints). + // Each entry is -1 (no replacement) or the global flat index of the best + // neighbor to copy from. std::vector sliceBestNeighbor(sliceSize, -1); - // Rolling window: slot 0 = z-1, slot 1 = z (current), slot 2 = z+1 - // Use unique_ptr instead of std::vector to avoid std::vector bit-packing + // Rolling window: 3 Z-slices of the input comparison array. + // Slot 0 = z-1, slot 1 = z (current), slot 2 = z+1. + // Uses unique_ptr instead of std::vector to avoid the + // std::vector bit-packing problem for boolean arrays. std::array, 3> inputSlices; for(auto& is : inputSlices) { @@ -142,9 +254,11 @@ struct ExecuteTemplate inputStore.copyIntoBuffer(zOffset, nonstd::span(inputSlices[slot].get(), sliceSize)); }; - // Face neighbor ordering: 0=-Z, 1=-Y, 2=-X, 3=+X, 4=+Y, 5=+Z + // Maps face-neighbor index to rolling-window slot: + // -Z -> slot 0, -Y/-X/+X/+Y -> slot 1 (same Z), +Z -> slot 2 constexpr std::array k_NeighborSlot = {0, 1, 1, 1, 1, 2}; + // ---- Main pass loop ---- while(keepGoing) { keepGoing = false; @@ -154,7 +268,7 @@ struct ExecuteTemplate break; } - // Initialize rolling window: load z=0 into slot 1, z=1 into slot 2 + // Re-initialize rolling window from the (potentially modified) store readInputSlice(0, 1); if(dims[2] > 1) { @@ -165,9 +279,10 @@ struct ExecuteTemplate int64 prog = 1; int64 progressInt = 0; + // ---- Z-slice scan loop ---- for(int64 zIdx = 0; zIdx < dims[2]; zIdx++) { - // Advance rolling window for z > 0 + // Advance the rolling window forward by one Z-slice if(zIdx > 0) { std::swap(inputSlices[0], inputSlices[1]); @@ -178,6 +293,7 @@ struct ExecuteTemplate } } + // ---- Inner XY scan ---- for(int64 yIdx = 0; yIdx < dims[1]; yIdx++) { for(int64 xIdx = 0; xIdx < dims[0]; xIdx++) @@ -185,6 +301,7 @@ struct ExecuteTemplate const int64 voxelIndex = zIdx * static_cast(sliceSize) + yIdx * dims[0] + xIdx; const usize inSlice = static_cast(yIdx * dims[0] + xIdx); + // Check if this voxel fails the threshold comparison if(comparator->compare(inputSlices[1][inSlice], ThresholdValue)) { count++; @@ -192,6 +309,7 @@ struct ExecuteTemplate std::array isValidFaceNeighbor = computeValidFaceNeighbors(xIdx, yIdx, zIdx, dims); + // Map each face neighbor to its in-slice offset const std::array neighborInSlice = { inSlice, // -Z static_cast((yIdx - 1) * dims[0] + xIdx), // -Y @@ -201,6 +319,7 @@ struct ExecuteTemplate inSlice // +Z }; + // Find the best qualifying neighbor among the 6 face neighbors for(const auto& faceIndex : faceNeighborInternalIdx) { if(!isValidFaceNeighbor[faceIndex]) @@ -223,7 +342,12 @@ struct ExecuteTemplate } } - // Transfer this Z-slice immediately (bestNeighbor only marks current voxel, not cross-slice) + // ---- Immediate per-slice transfer ---- + // Unlike ErodeDilateBadData where dilation marks cross Z-slices, + // this algorithm only marks the current voxel (the failing one) to + // receive data from one of its face neighbors. So the marks for + // each Z-slice are complete as soon as that slice's XY scan finishes. + // Transfer all sibling arrays in the Attribute Matrix. for(const auto& [dataId, dataObject] : *attrMatrix) { auto* dataArrayPtr = dynamic_cast(dataObject.get()); @@ -234,7 +358,7 @@ struct ExecuteTemplate SliceBufferedTransferOneZ(*dataArrayPtr, sliceBestNeighbor, sliceSize, static_cast(zIdx), dimZ); } - // Clear per-slice marks for next Z + // Clear per-slice marks for the next Z-slice std::fill(sliceBestNeighbor.begin(), sliceBestNeighbor.end(), -1); } @@ -243,6 +367,7 @@ struct ExecuteTemplate break; } + // If looping is enabled and modifications were made, schedule another pass if(loopUntilDone && count > 0) { keepGoing = true; @@ -279,6 +404,9 @@ Result<> ReplaceElementAttributesWithNeighborValues::operator()() auto* srcIDataArray = m_DataStructure.getDataAs(m_InputValues->InputArrayPath); const auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->SelectedImageGeometryPath); + // Dispatch to the type-specific inner loop. ExecuteDataFunction instantiates + // ExecuteTemplate::operator() with the correct element type T matching + // the input array's DataType (int32, float32, uint8, etc.). ExecuteDataFunction(ExecuteTemplate{}, srcIDataArray->getDataType(), imageGeom, srcIDataArray, m_InputValues->SelectedComparison, m_InputValues->MinConfidence, m_InputValues->Loop, m_ShouldCancel, m_MessageHandler); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp index beb606207e..25ecfdbff4 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReplaceElementAttributesWithNeighborValues.hpp @@ -22,19 +22,64 @@ inline const ChoicesParameter::Choices k_OperationChoices = {k_LessThan, k_Great /** * @struct ReplaceElementAttributesWithNeighborValuesInputValues * @brief Holds all user-supplied parameters for the ReplaceElementAttributesWithNeighborValues algorithm. + * + * Populated by the filter's preflight/execute methods and passed into the + * algorithm to decouple it from the parameter system. */ struct SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValuesInputValues { - float32 MinConfidence; - ChoicesParameter::ValueType SelectedComparison; - bool Loop; - DataPath InputArrayPath; - DataPath SelectedImageGeometryPath; + float32 MinConfidence; ///< Threshold value for the comparison (e.g., confidence index cutoff) + ChoicesParameter::ValueType SelectedComparison; ///< 0 = Less Than (replace voxels below threshold), 1 = Greater Than (replace voxels above threshold) + bool Loop; ///< If true, repeat until no voxels remain that fail the threshold test + DataPath InputArrayPath; ///< Path to the scalar cell array used for the threshold comparison + DataPath SelectedImageGeometryPath; ///< Path to the ImageGeom that defines the voxel grid }; /** * @class ReplaceElementAttributesWithNeighborValues - * @brief Replaces voxel data with the best face-neighbor value based on a threshold comparison. + * @brief Iteratively replaces voxel data that fails a threshold comparison with + * the best-scoring face-neighbor value, optimized for out-of-core (OOC) + * data stores. + * + * ## Algorithm Overview + * + * 1. For each voxel whose value fails the threshold test (less-than or + * greater-than the user's cutoff), examine its 6 face neighbors. + * 2. Among neighbors that pass the threshold, find the one with the + * best (highest or lowest, depending on comparison direction) value. + * 3. Mark that voxel to be replaced by the best neighbor's data. + * 4. After scanning a Z-slice, commit the marks by copying tuple data from + * source to destination for ALL arrays in the Attribute Matrix. + * 5. If Loop is true, repeat until no failing voxels remain. + * + * Unlike the ErodeDilateBadData algorithm (which only marks voxels within + * +/- 1 Z-slice via face neighbors), this algorithm's marks are strictly + * within the current Z-slice: a failing voxel is replaced by one of its + * own face neighbors, so the source is always within +/- 1 Z of the + * destination. This allows each Z-slice to be committed immediately after + * processing rather than being deferred. + * + * ## OOC Optimization Strategy + * + * 1. **3-slice rolling window for the input array**: Three Z-slices of the + * comparison array (prev, current, next) are held in typed buffers loaded + * via copyIntoBuffer(). All face-neighbor comparisons index into these + * buffers rather than the OOC store. + * + * 2. **Per-slice best-neighbor marks**: A single O(sliceSize) mark array + * tracks the best replacement source for each voxel in the current + * Z-slice. Marks are committed immediately via SliceBufferedTransferOneZ + * and then cleared for the next slice. + * + * 3. **Type-dispatched via ExecuteDataFunction**: The inner algorithm is + * templated on the input array's element type to avoid virtual dispatch + * overhead during the tight comparison loop. The template also handles + * the std::vector bit-packing issue by using unique_ptr for + * all buffer types. + * + * 4. **Per-pass re-read**: Each pass re-loads the rolling window from the + * store because the previous pass's SliceBufferedTransferOneZ calls + * may have changed the comparison array's values. */ class SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValues { @@ -60,18 +105,29 @@ class SIMPLNXCORE_EXPORT ReplaceElementAttributesWithNeighborValues ReplaceElementAttributesWithNeighborValues& operator=(ReplaceElementAttributesWithNeighborValues&&) noexcept = delete; /** - * @brief Executes the replace element attributes with neighbor values algorithm. + * @brief Executes the threshold-based neighbor replacement algorithm. + * + * Dispatches to a type-specific inner loop via ExecuteDataFunction. Each + * pass processes all Z-slices sequentially using a 3-slice rolling window, + * committing per-slice marks immediately via SliceBufferedTransferOneZ. + * Repeats until no failing voxels remain (if Loop is true) or for one + * pass (if Loop is false). + * * @return Result<> indicating success or any errors encountered during execution */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag. + * @return const reference to the atomic cancellation flag + */ const std::atomic_bool& getCancel() const; private: - DataStructure& m_DataStructure; - const ReplaceElementAttributesWithNeighborValuesInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure holding all arrays + const ReplaceElementAttributesWithNeighborValuesInputValues* m_InputValues = nullptr; ///< User-supplied algorithm parameters + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag checked between passes + const IFilter::MessageHandler& m_MessageHandler; ///< Handler for progress/status messages }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp index 23dd4b37aa..61fef26a14 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.cpp @@ -169,6 +169,17 @@ Result<> RequireMinimumSizeFeatures::operator()() return {}; } +/** + * @brief Iteratively fills voxels belonging to removed features (featureId < 0) + * by majority-voting among their 6 face-neighbors. + * + * OOC optimization: The voting scan uses a rolling 3-slice buffer for FeatureIds. + * For each Z-slice, the current slice and its Z-neighbors (prev, next) are in + * memory, so all 6 face-neighbor reads come from local buffers rather than + * per-element OOC DataStore access. The buffer slides forward one Z-slice per + * iteration. Only changed voxels are tracked and have their data arrays updated, + * rather than iterating over all voxels for each data array. + */ void RequireMinimumSizeFeatures::assignBadVoxels(SizeVec3 dimensions, const Int32AbstractDataStore& featureNumCellsStoreRef) { MessageHelper messageHelper(m_MessageHandler); @@ -203,8 +214,10 @@ void RequireMinimumSizeFeatures::assignBadVoxels(SizeVec3 dimensions, const Int3 { counter = 0; - // Scan phase: read featureIds in z-slice chunks for voting - // Use 3-slice rolling buffer so all 6 face neighbors are accessible + // Rolling 3-slice buffer: holds the previous, current, and next Z-slices so + // that all 6 face-neighbor reads come from local memory. The buffer is advanced + // one Z-slice at a time by swapping pointers and reading the next slice. + // This eliminates per-element OOC DataStore access during the voting scan. std::vector slabBuf(3 * sliceSize, 0); int32* prevSlice = slabBuf.data(); int32* curSlice = slabBuf.data() + sliceSize; @@ -381,6 +394,10 @@ std::vector RequireMinimumSizeFeatures::removeSmallFeatures(Int32AbstractD return activeObjects; } + // Mark removed features' voxels with featureId = -1 using chunked bulk I/O. + // The original per-element setValue(-1) caused a chunk operation per voxel. + // Reading/writing in 64K chunks reduces chunk operations by ~64000x. + // Only modified chunks are written back (tracked via the `modified` flag). auto featureIdBuf = std::make_unique(k_ChunkTuples); for(usize offset = 0; offset < totalPoints; offset += k_ChunkTuples) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.hpp index f1eea50e9f..1b0b89af71 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/RequireMinimumSizeFeatures.hpp @@ -12,22 +12,42 @@ namespace nx::core { +/** + * @struct RequireMinimumSizeFeaturesInputValues + * @brief Holds all user-configured parameters for the RequireMinimumSizeFeatures algorithm. + */ struct SIMPLNXCORE_EXPORT RequireMinimumSizeFeaturesInputValues { - BoolParameter::ValueType ApplySinglePhase; - ArraySelectionParameter::ValueType FeatureIdsPath; - ArraySelectionParameter::ValueType FeaturePhasesPath; - GeometrySelectionParameter::ValueType InputImageGeometryPath; - Int64Parameter::ValueType MinAllowedFeaturesSize; - ArraySelectionParameter::ValueType FeatureNumCellsPath; - Int32Parameter::ValueType PhaseNumber; + BoolParameter::ValueType ApplySinglePhase; ///< If true, only remove small features in one phase. + ArraySelectionParameter::ValueType FeatureIdsPath; ///< Per-cell Feature ID array (int32). + ArraySelectionParameter::ValueType FeaturePhasesPath; ///< Per-feature phase array (for single-phase mode). + GeometrySelectionParameter::ValueType InputImageGeometryPath; ///< Input ImageGeom. + Int64Parameter::ValueType MinAllowedFeaturesSize; ///< Minimum voxel count threshold. + ArraySelectionParameter::ValueType FeatureNumCellsPath; ///< Per-feature voxel count array. + Int32Parameter::ValueType PhaseNumber; ///< Phase to filter (when ApplySinglePhase is true). }; /** * @class RequireMinimumSizeFeatures - * @brief This algorithm implements support code for the RequireMinimumSizeFeaturesFilter + * @brief Removes features with fewer voxels than a user-specified minimum threshold, + * then iteratively fills the resulting gaps by voting among face-neighbor feature IDs. + * + * @section ooc_optimization Out-of-Core Optimization + * Two operations were optimized: + * + * **removeSmallFeatures()**: The original per-element setValue(-1) loop for marking + * removed features caused a chunk operation per voxel. The optimized version reads + * FeatureIds in 64K-tuple chunks via copyIntoBuffer(), modifies the buffer in-place, + * and writes back only modified chunks via copyFromBuffer(). + * + * **assignBadVoxels()**: The original per-element getValue() in the voting loop caused + * chunk thrashing across the entire volume on every iteration. The optimized version + * uses a rolling 3-slice buffer: for each Z-slice, the current slice and its Z-neighbors + * are in memory so all 6 face-neighbor reads come from local buffers. The buffer slides + * forward one slice at a time. Changed voxels are tracked in a compact list, and only + * those voxels have their data arrays updated (via copyTuple) rather than iterating + * over all voxels for each data array. */ - class SIMPLNXCORE_EXPORT RequireMinimumSizeFeatures { public: @@ -39,35 +59,42 @@ class SIMPLNXCORE_EXPORT RequireMinimumSizeFeatures RequireMinimumSizeFeatures& operator=(const RequireMinimumSizeFeatures&) = delete; RequireMinimumSizeFeatures& operator=(RequireMinimumSizeFeatures&&) noexcept = delete; + /** + * @brief Executes the minimum-size filter: removes small features, then fills gaps. + * @return Result<> indicating success or error. + */ Result<> operator()(); protected: /** - * - * @param dimensions - * @param featureNumCellsStoreRef + * @brief Iteratively fills voxels belonging to removed features (featureId < 0) + * by voting among their 6 face-neighbors. Uses a rolling 3-slice buffer to avoid + * per-element OOC access during the voting scan. + * @param dimensions XYZ dimensions of the ImageGeom. + * @param featureNumCellsStoreRef Per-feature voxel count array (for vote counter sizing). */ void assignBadVoxels(SizeVec3 dimensions, const Int32AbstractDataStore& featureNumCellsStoreRef); /** - * - * @param featureIdsStoreRef - * @param featureNumCellsStoreRef - * @param featurePhases - * @param phaseNumber - * @param applyToSinglePhase - * @param minAllowedFeatureSize - * @param errorReturn - * @return + * @brief Marks features below the minimum size as inactive and sets their voxels' + * Feature IDs to -1 using chunked bulk I/O. + * @param featureIdsStoreRef Per-cell Feature ID DataStore (modified in-place). + * @param featureNumCellsStoreRef Per-feature voxel count array. + * @param featurePhases Per-feature phase array (may be nullptr). + * @param phaseNumber Target phase number (when applyToSinglePhase is true). + * @param applyToSinglePhase If true, only remove features in the specified phase. + * @param minAllowedFeatureSize Minimum voxel count threshold. + * @param errorReturn Output: receives error details if all features would be removed. + * @return Vector of booleans indicating which features remain active. */ std::vector removeSmallFeatures(Int32AbstractDataStore& featureIdsStoreRef, const Int32AbstractDataStore& featureNumCellsStoreRef, const Int32AbstractDataStore* featurePhases, int32_t phaseNumber, bool applyToSinglePhase, int64 minAllowedFeatureSize, Error& errorReturn); private: - DataStructure& m_DataStructure; - const RequireMinimumSizeFeaturesInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the DataStructure. + const RequireMinimumSizeFeaturesInputValues* m_InputValues = nullptr; ///< User-configured parameters. + const std::atomic_bool& m_ShouldCancel; ///< Cancellation flag. + const IFilter::MessageHandler& m_MessageHandler; ///< Message handler for progress. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp index b8ae6fc94f..e7b0c13c1d 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ScalarSegmentFeatures.hpp @@ -16,23 +16,54 @@ namespace nx::core { +/** + * @struct ScalarSegmentFeaturesInputValues + * @brief Holds all user-configured parameters for the ScalarSegmentFeatures algorithm. + */ struct SIMPLNXCORE_EXPORT ScalarSegmentFeaturesInputValues { - int ScalarTolerance = 0; - bool UseMask; - bool RandomizeFeatureIds; - bool IsPeriodic = false; - SegmentFeatures::NeighborScheme NeighborScheme; - DataPath ImageGeometryPath; - DataPath InputDataPath; - DataPath MaskArrayPath; - DataPath FeatureIdsArrayPath; - DataPath CellFeatureAttributeMatrixPath; - DataPath ActiveArrayPath; + int ScalarTolerance = 0; ///< Maximum absolute difference between neighboring voxels for grouping. + bool UseMask; ///< If true, only voxels flagged as "good" in the mask participate. + bool RandomizeFeatureIds; ///< If true, randomize Feature IDs post-segmentation for visual contrast. + bool IsPeriodic = false; ///< If true, treat geometry boundaries as periodic (tileable). + SegmentFeatures::NeighborScheme NeighborScheme; ///< 6-face or 26-connected neighbor scheme. + DataPath ImageGeometryPath; ///< Path to the ImageGeom / IGridGeometry being segmented. + DataPath InputDataPath; ///< Path to the scalar array used for comparison (any numeric type). + DataPath MaskArrayPath; ///< Path to the boolean/uint8 mask array (used when UseMask is true). + DataPath FeatureIdsArrayPath; ///< Output: per-cell Feature ID array (int32). + DataPath CellFeatureAttributeMatrixPath; ///< Output: Attribute Matrix for per-feature arrays. + DataPath ActiveArrayPath; ///< Output: boolean array marking active features. }; /** - * @brief The ScalarSegmentFeatures class + * @class ScalarSegmentFeatures + * @brief Segments an ImageGeom into features by flood-filling contiguous voxels + * whose scalar values differ by no more than a user-specified tolerance. + * + * This is a general-purpose segmentation algorithm that works on any single-component + * scalar array (int8 through float64, plus boolean), unlike orientation-based + * segmentation filters (EBSD, CAxis). The tolerance defines the maximum absolute + * difference between neighboring voxels for them to be grouped into the same feature. + * + * @section dual_path Dual Algorithm Paths + * The algorithm has two execution paths selected automatically at runtime: + * - **In-core (DFS flood fill)**: Classic depth-first search via the base-class + * execute() method. Uses per-element access through the typed CompareFunctor. + * - **Out-of-core (CCL)**: Connected-component labeling via executeCCL(), which + * processes data Z-slice-by-Z-slice to limit memory usage. Activated when the + * FeatureIds array is backed by an OOC DataStore or ForceOocAlgorithm() is set. + * + * @section ooc_optimization Out-of-Core Optimization + * The CCL path uses a rolling 2-slot buffer system to avoid per-element virtual + * dispatch overhead on OOC DataStores. Before processing each Z-slice, prepareForSlice() + * bulk-reads the scalar input and mask arrays for that slice into contiguous + * in-memory buffers via copyIntoBuffer(). The isValidVoxel() and areNeighborsSimilar() + * overrides then read from these buffers instead of the underlying DataStore, + * eliminating chunk load/evict cycles. Two slots are needed because CCL compares + * the current slice with the previous slice (iz and iz-1). + * + * All scalar types are converted to float64 in the buffer for uniform comparison, + * and the tolerance is also cast to float64. */ class SIMPLNXCORE_EXPORT ScalarSegmentFeatures : public SegmentFeatures { @@ -48,21 +79,44 @@ class SIMPLNXCORE_EXPORT ScalarSegmentFeatures : public SegmentFeatures ScalarSegmentFeatures& operator=(const ScalarSegmentFeatures&) = delete; ScalarSegmentFeatures& operator=(ScalarSegmentFeatures&&) noexcept = delete; + /** + * @brief Executes the segmentation: sets up comparators, dispatches to DFS or CCL, + * then post-processes (resize AM, fill Active array, optionally randomize IDs). + * @return Result<> indicating success or error. + */ Result<> operator()(); protected: + /** + * @brief Finds the next unassigned voxel to seed a new feature (DFS path). + * @param gnum The feature number to assign to the seed. + * @param nextSeed Linear index to start scanning from. + * @return Linear index of the seed voxel, or -1 if no more seeds exist. + */ int64 getSeed(int32 gnum, int64 nextSeed) const override; + + /** + * @brief Determines whether a neighbor should be merged into the current feature (DFS path). + * Checks featureId == 0, mask validity, then delegates to the typed CompareFunctor. + * @param referencePoint Linear index of the reference voxel. + * @param neighborPoint Linear index of the candidate neighbor. + * @param gnum Current feature number (assigned to neighbor on success). + * @return true if the neighbor was merged into the feature. + */ bool determineGrouping(int64 referencePoint, int64 neighborPoint, int32 gnum) const override; /** - * @brief Checks whether a voxel can participate in scalar segmentation based on the mask. + * @brief Checks whether a voxel can participate in segmentation (CCL path). + * Uses the slice buffer fast path when available; falls back to direct MaskCompare access. * @param point Linear voxel index. * @return true if the voxel passes the mask check (or no mask is used). */ bool isValidVoxel(int64 point) const override; /** - * @brief Determines whether two neighboring voxels belong to the same scalar segment. + * @brief Determines whether two neighboring voxels have similar enough scalar values + * to belong to the same feature (CCL path). Uses the slice buffer fast path when + * both voxels' Z-slices are buffered; falls back to CompareFunctor otherwise. * @param point1 First voxel index. * @param point2 Second (neighbor) voxel index. * @return true if both voxels are valid and their scalar values are within tolerance. @@ -70,8 +124,13 @@ class SIMPLNXCORE_EXPORT ScalarSegmentFeatures : public SegmentFeatures bool areNeighborsSimilar(int64 point1, int64 point2) const override; /** - * @brief Pre-loads input scalar and mask data for the given Z-slice into - * rolling buffers, eliminating per-element OOC overhead during CCL. + * @brief Pre-loads input scalar and mask data for the given Z-slice into the + * rolling 2-slot buffer, eliminating per-element OOC overhead during CCL. + * + * Slot assignment: even slices go to slot 0, odd to slot 1. This ensures that + * both the current slice and the previous slice are always in memory. + * Passing iz = -1 disables buffering (used after the slice sweep for Phase 1b). + * * @param iz Current Z-slice index, or -1 to disable buffering. * @param dimX X dimension of the grid. * @param dimY Y dimension of the grid. @@ -80,21 +139,31 @@ class SIMPLNXCORE_EXPORT ScalarSegmentFeatures : public SegmentFeatures void prepareForSlice(int64 iz, int64 dimX, int64 dimY, int64 dimZ) override; private: + /** + * @brief Allocates the rolling 2-slot buffers for scalar and mask data. + * Each slot holds dimX * dimY elements (one full XY slice). + * @param dimX X dimension of the grid. + * @param dimY Y dimension of the grid. + */ void allocateSliceBuffers(int64 dimX, int64 dimY); + + /** + * @brief Releases the slice buffers and resets buffering state. + */ void deallocateSliceBuffers(); - const ScalarSegmentFeaturesInputValues* m_InputValues = nullptr; - FeatureIdsArrayType* m_FeatureIdsArray = nullptr; - GoodVoxelsArrayType* m_GoodVoxelsArray = nullptr; - std::shared_ptr m_CompareFunctor; - std::unique_ptr m_GoodVoxels = nullptr; - IDataArray* m_InputDataArray = nullptr; - - // Rolling 2-slot input buffers for OOC optimization. - std::vector m_ScalarBuffer; - std::vector m_MaskBuffer; - int64 m_BufSliceSize = 0; - int64 m_BufferedSliceZ[2] = {-1, -1}; - bool m_UseSliceBuffers = false; + const ScalarSegmentFeaturesInputValues* m_InputValues = nullptr; ///< User-configured parameters. + FeatureIdsArrayType* m_FeatureIdsArray = nullptr; ///< Output Feature IDs array. + GoodVoxelsArrayType* m_GoodVoxelsArray = nullptr; ///< Good voxels mask (if used). + std::shared_ptr m_CompareFunctor; ///< Typed comparator for DFS path. + std::unique_ptr m_GoodVoxels = nullptr; ///< Mask comparator. + IDataArray* m_InputDataArray = nullptr; ///< Raw pointer to the input scalar array (for type dispatch). + + // --- Rolling 2-slot input buffers for OOC optimization --- + std::vector m_ScalarBuffer; ///< Scalar values as float64 for uniform comparison (2 * sliceSize elements). + std::vector m_MaskBuffer; ///< Mask flags as uint8 (2 * sliceSize elements; 0 = masked out, 1 = valid). + int64 m_BufSliceSize = 0; ///< Number of voxels per XY slice (dimX * dimY). + int64 m_BufferedSliceZ[2] = {-1, -1}; ///< Z-index currently loaded in each slot (-1 = empty). + bool m_UseSliceBuffers = false; ///< True when the CCL path has activated slice buffering. }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp index 8476ed8943..12a1308faa 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.cpp @@ -1,3 +1,17 @@ +/** + * @file SurfaceNets.cpp + * @brief Dispatcher implementation for the SurfaceNets algorithm. + * + * This file contains the thin dispatch layer that examines the backing + * storage of the FeatureIds array and forwards execution to either: + * - SurfaceNetsDirect -- when all arrays are in-memory (uses MMSurfaceNet library) + * - SurfaceNetsScanline -- when any array uses chunked OOC storage + * + * The dispatch decision is made by DispatchAlgorithm, which inspects whether + * the DataStore is a chunked format. This avoids the severe performance + * penalty of random element access through virtual operator[] on OOC stores. + */ + #include "SurfaceNets.hpp" #include "SurfaceNetsDirect.hpp" #include "SurfaceNetsScanline.hpp" @@ -26,6 +40,15 @@ const std::atomic_bool& SurfaceNets::getCancel() } // ----------------------------------------------------------------------------- +/** + * @brief Dispatches to the correct algorithm variant based on DataStore type. + * + * The FeatureIds array is the primary input whose access pattern determines + * whether OOC optimization is needed. The Direct variant passes the raw + * DataStore to MMSurfaceNet which accesses it via operator[]. The Scanline + * variant reads FeatureIds via copyIntoBuffer() in Z-slices and builds its + * own O(surface) cell classification data structures. + */ Result<> SurfaceNets::operator()() { auto* featureIds = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp index 27a7c621ef..4326190a7f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNets.hpp @@ -10,32 +10,60 @@ namespace nx::core { +/** + * @struct SurfaceNetsInputValues + * @brief Aggregates every user-facing parameter and internally-created + * DataPath needed by the SurfaceNets algorithm family. + */ struct SIMPLNXCORE_EXPORT SurfaceNetsInputValues { - bool ApplySmoothing; - bool RepairTriangleWinding; - int32 SmoothingIterations; - float32 MaxDistanceFromVoxel; - float32 RelaxationFactor; - - DataPath GridGeomDataPath; - DataPath FeatureIdsArrayPath; - MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; - MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; - DataPath TriangleGeometryPath; - DataPath VertexGroupDataPath; - DataPath NodeTypesDataPath; - DataPath FaceGroupDataPath; - DataPath FaceLabelsDataPath; - MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; + bool ApplySmoothing; ///< When true, run iterative relaxation smoothing on mesh vertices + bool RepairTriangleWinding; ///< When true, run winding repair on the output triangle mesh + int32 SmoothingIterations; ///< Number of smoothing iterations to perform + float32 MaxDistanceFromVoxel; ///< Maximum distance a vertex can move from its voxel center during smoothing + float32 RelaxationFactor; ///< Blending factor for neighbor-averaging during smoothing (0..1) + + DataPath GridGeomDataPath; ///< Path to the input ImageGeom + DataPath FeatureIdsArrayPath; ///< Path to the Int32 FeatureIds cell array + MultiArraySelectionParameter::ValueType SelectedCellDataArrayPaths; ///< Cell arrays to transfer to the triangle face attribute matrix + MultiArraySelectionParameter::ValueType SelectedFeatureDataArrayPaths; ///< Feature arrays to transfer to the triangle face attribute matrix + DataPath TriangleGeometryPath; ///< Path to the created TriangleGeom output + DataPath VertexGroupDataPath; ///< Path to the vertex attribute matrix + DataPath NodeTypesDataPath; ///< Path to the Int8 NodeTypes vertex array + DataPath FaceGroupDataPath; ///< Path to the face attribute matrix + DataPath FaceLabelsDataPath; ///< Path to the Int32 FaceLabels (2-component) face array + MultiArraySelectionParameter::ValueType CreatedDataArrayPaths; ///< Paths to the created face arrays (transferred cell/feature data) }; /** - * @class + * @class SurfaceNets + * @brief Dispatcher that selects between SurfaceNetsDirect (in-core) and + * SurfaceNetsScanline (OOC) based on the storage type of input arrays. + * + * The Surface Nets algorithm generates a smoothed triangle mesh of feature + * boundaries using the method from Frisken (2022). Unlike QuickSurfaceMesh + * which places vertices at dual-grid corners, Surface Nets places vertices + * at voxel centers where features change, then optionally relaxes positions + * toward neighbor averages to produce smoother surfaces while preserving + * sharp boundaries between materials. + * + * Dispatch is performed by DispatchAlgorithm: if the FeatureIds array is + * backed by an in-memory DataStore, SurfaceNetsDirect is used (which + * delegates to the MMSurfaceNet library). If it uses chunked OOC storage, + * SurfaceNetsScanline is selected instead. + * + * @see SurfaceNetsDirect, SurfaceNetsScanline */ class SIMPLNXCORE_EXPORT SurfaceNets { public: + /** + * @brief Constructs the dispatcher. + * @param dataStructure The DataStructure containing all input/output objects + * @param mesgHandler Callback for progress and status messages + * @param shouldCancel Atomic flag checked periodically for user cancellation + * @param inputValues Pointer to the parameter struct (must outlive this object) + */ SurfaceNets(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, SurfaceNetsInputValues* inputValues); ~SurfaceNets() noexcept; @@ -44,15 +72,22 @@ class SIMPLNXCORE_EXPORT SurfaceNets SurfaceNets& operator=(const SurfaceNets&) = delete; SurfaceNets& operator=(SurfaceNets&&) noexcept = delete; + /** + * @brief Dispatches to the appropriate in-core or OOC algorithm implementation. + * @return Result<> indicating success or an error code from the selected algorithm + */ Result<> operator()(); + /** + * @brief Returns a reference to the cancellation flag (used by MMSurfaceNet internals). + */ const std::atomic_bool& getCancel(); private: - DataStructure& m_DataStructure; - const SurfaceNetsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the active DataStructure + const SurfaceNetsInputValues* m_InputValues = nullptr; ///< User parameters and created array paths + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message callback }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp index baae3c762a..db9a833ebf 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.cpp @@ -1,3 +1,37 @@ +/** + * @file SurfaceNetsDirect.cpp + * @brief In-core implementation of the SurfaceNets algorithm using the MMSurfaceNet library. + * + * This file delegates cell classification, vertex placement, and optional + * smoothing to the MMSurfaceNet library, which operates on the full padded + * grid in memory. The algorithm then extracts quads from edge crossings, + * triangulates them, and writes the output TriangleGeom. + * + * The MMSurfaceNet library accesses the FeatureIds array via operator[], which + * is efficient for in-memory DataStores but would cause severe chunk thrashing + * on OOC stores. This is why SurfaceNetsScanline exists as an alternative. + * + * ## Phases + * + * Phase 1: MMSurfaceNet constructs a padded grid (dimX+2, dimY+2, dimZ+2) and + * classifies each cell by examining its 8 corner labels. Surface cells + * (where not all corners match) get vertices at cell centers. + * + * Phase 2: Optional smoothing via MMSurfaceNet::relax() moves vertices toward + * neighbor averages, clamped to MaxDistanceFromVoxel. + * + * Phase 3: Vertex positions are transformed from local cell coordinates to + * world coordinates using ImageGeom origin and spacing. + * + * Phase 4: First pass counts triangles by checking 3 edges per surface vertex. + * + * Phase 5: Second pass generates triangle connectivity and face labels. + * Each edge crossing produces a quad (4 vertices), which is split + * into 2 triangles using the minimal-area diagonal. + * + * Phase 6: Optional winding repair. + */ + #include "SurfaceNetsDirect.hpp" #include "SurfaceNets.hpp" @@ -23,11 +57,20 @@ using namespace nx::core; namespace { using LabelType = int32; +/** + * @brief Adjusts a node type value for vertices on the exterior boundary. + * Values < 10 get +10 added to indicate they are boundary nodes. + * Values >= 10 are already marked as boundary and get +1. + */ constexpr inline int8 CalculatePadding(int8 value) { return value + ((9 * static_cast(value < 10)) + 1); } +/** + * @brief Marks all 4 vertices of a quad as boundary nodes when one of the + * quad's labels is MMSurfaceNet::Padding (i.e., the exterior of the volume). + */ inline void HandlePadding(std::array vertexIndices, AbstractDataStore& nodeTypes) { nodeTypes.setValue(vertexIndices[0], CalculatePadding(nodeTypes.getValue(vertexIndices[0]))); @@ -60,9 +103,22 @@ float32 triangleArea(std::array& vert0, std::array& vert return 0.5f * magCP; } +/** + * @brief Splits a quad into 2 triangles with consistent winding and minimal area. + * + * The quad is defined by 4 vertices in vData. The function: + * 1. Flips winding if the quad is back-facing (swaps vertices 1 and 3) + * 2. Chooses the triangulation diagonal that minimizes total triangle area, + * which reduces self-intersections in the resulting mesh + * 3. Writes 6 vertex IDs into triangleVtxIDs (2 triangles x 3 vertices) + * + * @param[in,out] vData The 4 quad vertices (may be reordered for winding/area) + * @param isQuadFrontFacing Whether labels[0] < labels[1] (determines initial winding) + * @param[out] triangleVtxIDs 6 vertex IDs forming 2 triangles + */ void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing, std::array& triangleVtxIDs) { - // Order quad vertices so quad is front facing + // Step 1: Ensure consistent front-facing winding by swapping vertices 1 and 3 if(!isQuadFrontFacing) { VertexData const temp = vData[3]; @@ -70,8 +126,8 @@ void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing vData[1] = temp; } - // Order quad vertices so that the two generated triangles have the minimal area. This - // reduces self intersections in the surface. + // Step 2: Choose the triangulation diagonal (0-2 vs 1-3) that minimizes + // total triangle area, reducing self-intersections in the surface mesh float32 const thisArea = triangleArea(vData[0].Position, vData[1].Position, vData[2].Position) + triangleArea(vData[0].Position, vData[2].Position, vData[3].Position); float32 const alternateArea = triangleArea(vData[1].Position, vData[2].Position, vData[3].Position) + triangleArea(vData[1].Position, vData[3].Position, vData[0].Position); if(alternateArea < thisArea) @@ -83,7 +139,7 @@ void getQuadTriangleIDs(std::array& vData, bool isQuadFrontFacing vData[3] = temp; } - // Generate vertex ids to triangulate the quad + // Step 3: Output the 2 triangles from the quad (fan triangulation from vData[0]) triangleVtxIDs[0] = vData[0].VertexId; triangleVtxIDs[1] = vData[1].VertexId; triangleVtxIDs[2] = vData[2].VertexId; @@ -106,14 +162,20 @@ SurfaceNetsDirect::SurfaceNetsDirect(DataStructure& dataStructure, const IFilter SurfaceNetsDirect::~SurfaceNetsDirect() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Executes the full in-core Surface Nets pipeline. + * + * Delegates cell classification and smoothing to the MMSurfaceNet library, + * then extracts edge-crossing quads and triangulates them. The MMSurfaceNet + * constructor reads the entire FeatureIds array via operator[], which is + * efficient for in-memory stores but would thrash on OOC stores. + */ Result<> SurfaceNetsDirect::operator()() { MessageHelper messageHelper(m_MessageHandler); - // Get the ImageGeometry auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); - // Get the Created Triangle Geometry auto& triangleGeom = m_DataStructure.getDataRefAs(m_InputValues->TriangleGeometryPath); auto* triangleGeomPtr = m_DataStructure.getDataAs(m_InputValues->TriangleGeometryPath); @@ -121,6 +183,10 @@ Result<> SurfaceNetsDirect::operator()() auto voxelSize = imageGeom.getSpacing(); auto origin = imageGeom.getOrigin(); + // Phase 1: Build the surface net. MMSurfaceNet classifies every cell in a + // padded grid (dim+2 in each direction) by examining the 8 corner labels. + // Cells where not all corners match get a vertex at the cell center. + // This reads the entire FeatureIds array via operator[] -- fast in-core only. messageHelper.sendMessage("Phase 1: Building surface net..."); MMSurfaceNet surfaceNet(triangleGeomPtr->getVerticesRef().getDataStoreRef(), m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath), gridDimensions.data(), voxelSize.data()); if(!surfaceNet.getCellMap()->valid()) @@ -133,7 +199,9 @@ Result<> SurfaceNetsDirect::operator()() return {}; } - // Use current parameters to relax the SurfaceNet + // Phase 2: Optional smoothing -- iterative Laplacian-like relaxation that + // moves each vertex toward the average of its face-connected neighbors, + // clamped to stay within MaxDistanceFromVoxel of the cell center. if(m_InputValues->ApplySmoothing) { messageHelper.sendMessage("Phase 2: Smoothing surface net..."); @@ -157,6 +225,9 @@ Result<> SurfaceNetsDirect::operator()() auto& nodeTypes = m_DataStructure.getDataAs(m_InputValues->NodeTypesDataPath)->getDataStoreRef(); nodeTypes.resizeTuples({static_cast(nodeCount)}); + // Phase 3: Transform vertex positions from local cell-relative coordinates + // (where 0.5 = cell center) to world coordinates using origin and spacing. + // Also assigns node types from the MMCellFlag junction count. messageHelper.sendMessage("Phase 3: Transforming vertex positions..."); Point3Df position = {0.0f, 0.0f, 0.0f}; @@ -177,10 +248,14 @@ Result<> SurfaceNetsDirect::operator()() nodeTypes[static_cast(vertIndex)] = static_cast(currentCellPtr->flag.numJunctions()); } + // Phase 4: Count triangles by checking 3 edges per surface vertex. + // Each cell has 12 edges, but by convention only 3 are checked per vertex + // (BackBottom, LeftBottom, LeftBack), which ensures each edge is counted + // exactly once across the grid. Each edge crossing produces a quad = 2 triangles. messageHelper.sendMessage("Phase 4: Counting triangles..."); usize triangleCount = 0; std::array quadNxArrayIndices = {0, 0}; - // First Pass through to just count the number of triangles: + // First pass: count only, do not write any mesh data for(int idxVtx = 0; idxVtx < nodeCount; idxVtx++) { if(m_ShouldCancel) @@ -241,11 +316,15 @@ Result<> SurfaceNetsDirect::operator()() ::AddFeatureTupleTransferInstance(m_DataStructure, selectedPath, createdPath, m_InputValues->FeatureIdsArrayPath, tupleTransferFunctions); } + // Phase 5: Generate triangles. Same edge iteration as Phase 4, but now + // writes triangle connectivity, face labels, and runs TupleTransfer. + // Each edge crossing produces a quad defined by 4 vertices from the current + // cell and 3 neighboring cells. The quad is split into 2 triangles using + // the minimal-area diagonal via getQuadTriangleIDs(). messageHelper.sendMessage("Phase 5: Generating triangles..."); usize faceIndex = 0; - // Create temporary storage for cell quads which are constructed around edges - // crossed by the surface. Handle 3 edges per cell. The other 9 cell edges will - // be handled when neighboring cells that share edges with this cell are visited. + // Handle 3 edges per cell (BackBottom, LeftBottom, LeftBack). The other 9 + // cell edges are handled when neighboring cells that share those edges are visited. std::array t1 = {0, 0, 0}; std::array t2 = {0, 0, 0}; std::array triangleVtxIDs = {0, 0, 0, 0, 0, 0}; @@ -438,7 +517,8 @@ Result<> SurfaceNetsDirect::operator()() } } - // Now run through the FaceLabels to make them consistent with Quick Surface Mesh + // Replace Padding label (0) with -1 to match QuickSurfaceMesh convention + // where -1 indicates the exterior of the volume for(usize tIdx = 0; tIdx < triangleCount * 2; tIdx++) { if(faceLabels[tIdx] == 0) diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp index 11a4d3db64..8432ea8bca 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsDirect.hpp @@ -11,13 +11,50 @@ struct SurfaceNetsInputValues; /** * @class SurfaceNetsDirect - * @brief In-core algorithm for SurfaceNets. Preserves the original sequential - * voxel iteration using MMSurfaceNet. Selected by DispatchAlgorithm when all - * input arrays are backed by in-memory DataStore. + * @brief In-core algorithm for SurfaceNets that delegates to the MMSurfaceNet + * library for cell classification and mesh generation. + * + * Selected by DispatchAlgorithm when all input arrays are backed by in-memory + * DataStore. This is the original algorithm implementation and serves as the + * reference for correctness. + * + * The algorithm runs in six phases: + * 1. **Build surface net** -- MMSurfaceNet classifies every cell in a padded + * grid (dimX+2, dimY+2, dimZ+2), identifying surface cells where the 8 + * corner labels are not all identical. Each surface cell gets a vertex. + * Uses operator[] to read FeatureIds, which is fast for in-memory stores. + * + * 2. **Smooth surface net** (optional) -- Iterative relaxation moves vertices + * toward the average of their face-connected neighbors, clamped to stay + * within MaxDistanceFromVoxel of their cell center. + * + * 3. **Transform vertices** -- Converts local cell-relative positions to + * world coordinates using the ImageGeom origin and spacing. + * + * 4. **Count triangles** -- Iterates surface vertices, checking 3 edges per + * cell (BackBottom, LeftBottom, LeftBack) for crossings that produce quads. + * Each quad becomes 2 triangles. + * + * 5. **Generate triangles** -- Second pass that writes triangle connectivity, + * face labels, and runs TupleTransfer for cell/feature data. + * + * 6. **Winding repair** (optional) -- Fixes inconsistent triangle orientations. + * + * Memory: O(volume) for the MMCellMap internal data structure (one Cell per + * padded voxel). This is the main reason the Scanline variant exists. + * + * @see SurfaceNetsScanline for the OOC-optimized variant */ class SIMPLNXCORE_EXPORT SurfaceNetsDirect { public: + /** + * @brief Constructs the in-core algorithm. + * @param dataStructure The DataStructure containing all input/output objects + * @param mesgHandler Callback for progress and status messages + * @param shouldCancel Atomic flag checked periodically for user cancellation + * @param inputValues Pointer to the parameter struct (must outlive this object) + */ SurfaceNetsDirect(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const SurfaceNetsInputValues* inputValues); ~SurfaceNetsDirect() noexcept; @@ -26,13 +63,19 @@ class SIMPLNXCORE_EXPORT SurfaceNetsDirect SurfaceNetsDirect& operator=(const SurfaceNetsDirect&) = delete; SurfaceNetsDirect& operator=(SurfaceNetsDirect&&) noexcept = delete; + /** + * @brief Executes the full in-core Surface Nets pipeline: cell classification, + * optional smoothing, vertex transformation, triangle generation, and optional + * winding repair. + * @return Result<> indicating success or an error from allocation or winding repair + */ Result<> operator()(); private: - DataStructure& m_DataStructure; - const SurfaceNetsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the active DataStructure + const SurfaceNetsInputValues* m_InputValues = nullptr; ///< User parameters and created array paths + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message callback }; } // namespace nx::core diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp index 73c4e6a81a..884198533c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.cpp @@ -1,3 +1,45 @@ +/** + * @file SurfaceNetsScanline.cpp + * @brief Out-of-core (OOC) optimized implementation of the SurfaceNets algorithm. + * + * This file reimplements the Surface Nets algorithm without using the MMSurfaceNet + * library (which requires O(volume) memory and per-element FeatureIds access). + * Instead, it reads FeatureIds via copyIntoBuffer() in Z-slice pairs and stores + * only the O(surface) cells that actually have vertices. + * + * ## Key Differences from SurfaceNetsDirect + * + * 1. **No MMSurfaceNet/MMCellMap**: The Direct variant delegates cell classification + * to MMSurfaceNet, which allocates a Cell for every padded voxel (O(volume)). + * The Scanline variant performs cell classification inline using MMCellFlag + * directly, storing only surface cells in a vector + hash map (O(surface)). + * + * 2. **FeatureIds access**: Direct passes the full array to MMSurfaceNet which + * reads via operator[]. Scanline reads two Z-slices at a time via + * copyIntoBuffer() and resolves cell corner labels from the buffered slices + * using the cornerLabel() helper. + * + * 3. **Smoothing**: Direct delegates to MMSurfaceNet::relax() which operates on + * the full MMCellMap. Scanline performs smoothing on the O(surface) m_Vertices + * array using neighbor lookups through the m_CellToVertex hash map. + * + * 4. **Edge quad generation**: Direct uses MMCellMap::getEdgeQuad() which looks + * up neighboring cells in the O(volume) cell array. Scanline uses + * lookupVertex() to find neighbors in the O(surface) hash map. + * + * 5. **Output writes**: Direct writes per-element. Scanline buffers all output + * (triangle connectivity, face labels, vertex positions, node types) and + * flushes via copyFromBuffer() in bulk. + * + * ## Memory Comparison + * + * For a 500x500x500 dataset with ~1% surface cells: + * - Direct (MMCellMap): ~500^3 * sizeof(Cell) = O(125M) entries + * - Scanline: ~500^3 * 0.01 = O(1.25M) entries + hash map overhead + * + * The Scanline variant uses roughly 100x less memory for cell classification. + */ + #include "SurfaceNetsScanline.hpp" #include "SurfaceNets.hpp" @@ -34,11 +76,19 @@ inline uint64 packCellKey(int32 i, int32 j, int32 k, int32 paddedX, int32 padded return static_cast(i) + static_cast(j) * static_cast(paddedX) + static_cast(k) * static_cast(paddedXY); } +/** + * @brief Adjusts a node type value for vertices on the exterior boundary. + * Same logic as SurfaceNetsDirect but operates on the local buffer. + */ constexpr inline int8 CalculatePadding(int8 value) { return value + ((9 * static_cast(value < 10)) + 1); } +/** + * @brief Marks all 4 vertices of a quad as boundary nodes in the local buffer + * when one of the quad's labels is MMSurfaceNet::Padding. + */ inline void HandlePadding(std::array vertexIndices, std::vector& nodeTypesBuf) { nodeTypesBuf[vertexIndices[0]] = CalculatePadding(nodeTypesBuf[vertexIndices[0]]); @@ -224,10 +274,37 @@ SurfaceNetsScanline::SurfaceNetsScanline(DataStructure& dataStructure, const IFi SurfaceNetsScanline::~SurfaceNetsScanline() noexcept = default; // ----------------------------------------------------------------------------- +/** + * @brief Executes the full OOC Surface Nets pipeline. + * + * This method is the most complex OOC algorithm in the codebase because it + * reimplements the entire Surface Nets algorithm from scratch without the + * MMSurfaceNet library, using only O(surface) memory and bulk I/O. + * + * The method is organized into numbered sections (1-13) that correspond to + * the phases documented in the header. The sections are: + * + * 1-4. Cell classification: iterate padded grid, load Z-slice pairs, + * classify cells using MMCellFlag, store surface cells + * 5. Resize vertex arrays + * 6. Phase 2A: Optional smoothing using O(surface) data structures + * 7. Phase 2B: Transform to world coordinates, assign node types + * 8. Phase 3A: Count triangles from edge crossings + * 9. Phase 3B: Resize face arrays + * 10. Phase 3C: Setup TupleTransfer + * 11. Phase 3D: Generate triangles and buffer output + * 12. Phase 3E: Fix face labels (0 -> -1) + * 12b. Flush all buffered output via copyFromBuffer() + * 13. Phase 3F: Optional winding repair + */ Result<> SurfaceNetsScanline::operator()() { // ------------------------------------------------------------------------- - // 1. Get ImageGeom dimensions and compute padded dims + // 1. Get ImageGeom dimensions and compute padded dims. + // The padded grid adds 1 cell of padding on each side in each dimension, + // matching the convention used by MMSurfaceNet. Padding cells always have + // the MMSurfaceNet::Padding label, which creates boundary triangles at the + // volume edges. // ------------------------------------------------------------------------- auto& imageGeom = m_DataStructure.getDataRefAs(m_InputValues->GridGeomDataPath); auto gridDimensions = imageGeom.getDimensions(); @@ -247,21 +324,37 @@ Result<> SurfaceNetsScanline::operator()() auto& featureIdsStore = m_DataStructure.getDataAs(m_InputValues->FeatureIdsArrayPath)->getDataStoreRef(); // ------------------------------------------------------------------------- - // 3. Allocate rolling slice buffers (2 NX Z-slices) + // 3. Allocate rolling slice buffers (2 NX Z-slices). + // Each buffer holds one full XY plane of FeatureIds. The ping-pong scheme + // ensures at most 2 Z-slices are in memory at once. The slice0Z/slice1Z + // variables track which NX Z-index each buffer currently holds, to avoid + // re-reading a slice that is already buffered. // ------------------------------------------------------------------------- const usize sliceSize = dimX * dimY; std::vector sliceBufA(sliceSize); std::vector sliceBufB(sliceSize); - // Pointers for ping-pong: slice0 is the "lower" Z-slice, slice1 is "upper" + // Pointers for ping-pong: slice0 is the "lower" Z-slice, slice1 is "upper". + // These are swapped at the end of each k iteration so the "upper" becomes + // "lower" for the next iteration. std::vector* slice0 = &sliceBufA; std::vector* slice1 = &sliceBufB; - int32 slice0Z = -1; - int32 slice1Z = -1; + int32 slice0Z = -1; // NX Z-index currently in slice0 (-1 = empty) + int32 slice1Z = -1; // NX Z-index currently in slice1 (-1 = empty) // ------------------------------------------------------------------------- - // 4. Iterate cells in the same order as setCellVertices(): - // k in [0, paddedZ-2), j in [0, paddedY-2), i in [0, paddedX-2) + // 4. Phase 1: Cell classification. + // Iterate cells in the same order as MMCellMap::setCellVertices(): + // k in [0, paddedZ-2), j in [0, paddedY-2), i in [0, paddedX-2) + // + // For each cell, compute the 8 corner labels using cornerLabel(), which + // reads from the buffered Z-slices. The corner ordering matches MMCellMap: + // [0]=(i,j,k) [1]=(i+1,j,k) [2]=(i+1,j+1,k) [3]=(i,j+1,k) + // [4]=(i,j,k+1) [5]=(i+1,j,k+1) [6]=(i+1,j+1,k+1) [7]=(i,j+1,k+1) + // + // MMCellFlag::set() classifies the cell: if all 8 labels are identical, + // it is an interior cell (no vertex). Otherwise it is a surface cell + // and gets stored in m_Vertices with its padded coordinates and flag. // ------------------------------------------------------------------------- const usize totalPaddedZSlices = static_cast(paddedZ - 1); for(int32 k = 0; k < paddedZ - 1; k++) @@ -363,10 +456,23 @@ Result<> SurfaceNetsScanline::operator()() std::vector nodeTypesBuf(numVertices, 0); // ------------------------------------------------------------------------- - // 6. Phase 2A: Relaxation (optional — only if smoothing is requested) + // 6. Phase 2A: Relaxation (optional -- only if smoothing is requested). + // + // This reimplements MMSurfaceNet::relax() using the O(surface) m_Vertices + // and m_CellToVertex data structures instead of the O(volume) MMCellMap. + // + // For each surface vertex, the relaxation: + // 1. Finds up to 6 face-connected neighbors via m_CellToVertex lookup + // 2. Computes the average neighbor position (in cell-local coordinates) + // 3. Blends current position toward the average: p' = (1-alpha)*p + alpha*avg + // 4. Clamps to [0.5 - maxDist, 0.5 + maxDist] to keep vertices near cell centers + // + // The face participation rule depends on vertex type: + // - SurfaceVertex: participates if the face has any crossing + // - JunctionVertex: participates only if the face has a junction crossing + // + // Face offsets define the 6 directions to check for neighbors. // ------------------------------------------------------------------------- - // Face direction offsets: Left(-1,0,0), Right(+1,0,0), Back(0,-1,0), - // Front(0,+1,0), Bottom(0,0,-1), Top(0,0,+1) static constexpr int32 k_FaceOffsets[6][3] = { {-1, 0, 0}, // LeftFace {+1, 0, 0}, // RightFace @@ -376,8 +482,9 @@ Result<> SurfaceNetsScanline::operator()() {0, 0, +1} // TopFace }; - // Use a local buffer for all position work. Initial value is (0.5, 0.5, 0.5) - // -- cell center in local coords. This avoids O(iterations * vertices * 6) + // Local position buffer: each vertex starts at (0.5, 0.5, 0.5) which is the + // cell center in local coordinates. Smoothing modifies these in-place. + // After smoothing, Phase 2B transforms them to world coordinates. std::vector localPos(3 * numVertices); for(usize v = 0; v < numVertices; v++) { @@ -485,12 +592,23 @@ Result<> SurfaceNetsScanline::operator()() // ------------------------------------------------------------------------- // 7. Phase 2B: Transform vertex positions to world coordinates and assign - // NodeTypes. Replicates SurfaceNetsDirect.cpp lines 148-161. + // NodeTypes. + // + // Converts cell-local coordinates (where 0.5 = cell center) to world + // coordinates using the formula: + // worldPos = voxelSize * (cellIndex + localPos) + origin - halfVoxelOffset + // + // The halfVoxelOffset accounts for the fact that padded cell index 1 + // corresponds to the first NX voxel, whose center is at origin + 0.5*spacing. + // + // Node types are derived from MMCellFlag::numJunctions(): the number of + // distinct material junctions at the vertex. + // + // Note: the Z component of halfVoxelOffset intentionally uses voxelSize[1] + // to match the original SurfaceNetsDirect.cpp behavior (line 155). // ------------------------------------------------------------------------- auto voxelSize = imageGeom.getSpacing(); auto origin = imageGeom.getOrigin(); - // Note: the Z offset intentionally uses voxelSize[1] to match the original - // SurfaceNetsDirect.cpp behavior (line 155). const Point3Df halfVoxelOffset(0.5f * voxelSize[0], 0.5f * voxelSize[1], 0.5f * voxelSize[1]); // Build vertex positions in localPos (reusing the smoothing buffer) and @@ -511,7 +629,21 @@ Result<> SurfaceNetsScanline::operator()() } // ------------------------------------------------------------------------- - // 8. Phase 3A: First pass — count triangles + // 8. Phase 3A: First pass -- count triangles. + // + // Iterates all surface vertices and checks 3 edges per cell: + // - BackBottomEdge: between this cell and (i+1, j, k) + // - LeftBottomEdge: between this cell and (i, j+1, k) + // - LeftBackEdge: between this cell and (i, j, k+1) + // + // Each edge crossing produces a quad = 2 triangles. The 4 quad vertices + // are the current cell's vertex plus 3 neighbors found via lookupVertex(). + // + // This pass also marks boundary vertices via HandlePadding() when one of + // the quad labels is Padding (exterior of volume). + // + // After this pass, vertex positions and node types are flushed to their + // DataStores via copyFromBuffer() since they are now finalized. // ------------------------------------------------------------------------- usize triangleCount = 0; std::array quadNxArrayIndices = {0, 0}; @@ -612,11 +744,25 @@ Result<> SurfaceNetsScanline::operator()() } // ------------------------------------------------------------------------- - // 11. Phase 3D: Second pass — generate triangles + // 11. Phase 3D: Second pass -- generate triangles. + // + // Same edge iteration as Phase 3A, but now writes triangle data. All + // output is accumulated in local buffers: + // - triConnBuf: triangle connectivity (3 MeshIndexType per triangle) + // - faceLabelBuf: face labels (2 int32 per triangle) + // - ttArgsBuf: TupleTransfer arguments (1 per triangle) + // + // These are flushed in one bulk write at the end (section 12b). This + // avoids per-element writes to OOC DataStores. + // + // Each edge crossing produces a quad from 4 vertices, which is split into + // 2 triangles using getQuadTriangleIDs() (minimal-area diagonal selection). + // Face labels are ordered so the smaller label is first. // ------------------------------------------------------------------------- using MeshIndexType = IGeometry::MeshIndexType; auto& facesStore = triangleGeom.getFacesRef().getDataStoreRef(); + // Pre-allocate output buffers for all triangles (known count from Phase 3A) std::vector triConnBuf; std::vector faceLabelBuf; std::vector ttArgsBuf; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp index 488d65d0a1..b11421d1f5 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/SurfaceNetsScanline.hpp @@ -17,16 +17,74 @@ struct SurfaceNetsInputValues; /** * @class SurfaceNetsScanline - * @brief Out-of-core algorithm for SurfaceNets. Selected by DispatchAlgorithm - * when any input array is backed by chunked (OOC) storage. + * @brief Out-of-core (OOC) optimized algorithm for SurfaceNets. * - * Phase 1 performs slice-by-slice cell classification, reading FeatureIds - * via copyIntoBuffer in Z-slices. Surface cells are stored in O(surface) - * data structures rather than the O(n) Cell array used by MMCellMap. + * Selected by DispatchAlgorithm when any input array is backed by chunked + * (OOC) storage. Produces identical output to SurfaceNetsDirect but avoids + * the O(volume) MMCellMap allocation and per-element FeatureIds access. + * + * ## OOC Strategy + * + * The key insight is that the Surface Nets cell classification only needs + * the 8 corner labels of each cell, which span at most 2 adjacent Z-slices. + * The scanline variant exploits this by: + * + * - **Bulk I/O**: Reading FeatureIds two Z-slices at a time via + * copyIntoBuffer() with a rolling ping-pong buffer. Each cell's 8 + * corner labels are resolved from the two buffered slices. + * + * - **O(surface) storage**: Instead of allocating one Cell per padded + * voxel (the O(volume) MMCellMap), this algorithm stores only the + * surface cells -- those where not all 8 corner labels are identical. + * Surface cells are stored in m_Vertices (a vector of VertexInfo) and + * m_CellToVertex (a hash map from padded cell coordinate to vertex index). + * For typical datasets, surface is O(n^{2/3}) vs O(n) for volume. + * + * - **Self-contained smoothing**: The relaxation is performed entirely on + * the O(surface) vertex data using neighbor lookups through m_CellToVertex, + * without needing the full MMCellMap. + * + * - **Buffered output writes**: Triangle connectivity, face labels, and + * vertex coordinates are accumulated in local buffers and flushed via + * copyFromBuffer() in bulk. + * + * ## Phases + * + * 1. **Cell classification** (operator() main loop) -- Iterates padded cells + * in Z-slice order, reads 8 corner labels from rolling slice buffers, + * computes MMCellFlag for each cell, stores surface cells in m_Vertices. + * + * 2A. **Smoothing** (optional) -- Iterative relaxation using face-connected + * neighbor positions looked up from m_CellToVertex. Same convergence + * behavior as MMSurfaceNet::relax() but operates on O(surface) data. + * + * 2B. **Vertex transform** -- Converts local cell-relative positions to + * world coordinates and assigns node types from MMCellFlag junction counts. + * + * 3A. **Triangle counting** -- Iterates surface vertices checking 3 edges + * per cell for crossings. Each crossing produces a quad = 2 triangles. + * + * 3B-3E. **Triangle generation** -- Second pass writes triangle connectivity, + * face labels, and runs TupleTransfer. All output is buffered and flushed + * in bulk via copyFromBuffer(). + * + * 3F. **Winding repair** (optional) -- Same as SurfaceNetsDirect. + * + * Memory: O(surface_cells) for m_Vertices + O(surface_cells) for m_CellToVertex + * + O(triangle_count * 3) for output buffers. + * + * @see SurfaceNetsDirect for the in-core reference implementation */ class SIMPLNXCORE_EXPORT SurfaceNetsScanline { public: + /** + * @brief Constructs the OOC-optimized algorithm. + * @param dataStructure The DataStructure containing all input/output objects + * @param mesgHandler Callback for progress and status messages + * @param shouldCancel Atomic flag checked periodically for user cancellation + * @param inputValues Pointer to the parameter struct (must outlive this object) + */ SurfaceNetsScanline(DataStructure& dataStructure, const IFilter::MessageHandler& mesgHandler, const std::atomic_bool& shouldCancel, const SurfaceNetsInputValues* inputValues); ~SurfaceNetsScanline() noexcept; @@ -35,25 +93,39 @@ class SIMPLNXCORE_EXPORT SurfaceNetsScanline SurfaceNetsScanline& operator=(const SurfaceNetsScanline&) = delete; SurfaceNetsScanline& operator=(SurfaceNetsScanline&&) noexcept = delete; + /** + * @brief Executes the full OOC Surface Nets pipeline: cell classification, + * optional smoothing, vertex transformation, triangle generation, and optional + * winding repair. + * @return Result<> indicating success or an error from winding repair + */ Result<> operator()(); /** * @brief Per-vertex information stored only for surface cells. + * + * This struct replaces the full MMCellMap::Cell for surface cells. It stores + * the padded grid coordinates and the MMCellFlag that encodes which edges + * and faces of the cell are crossed by the feature boundary. */ struct VertexInfo { - std::array cellIndex; // (i,j,k) in padded coordinates - MMCellFlag flag; + std::array cellIndex; ///< (i,j,k) position in the padded coordinate system + MMCellFlag flag; ///< Cell classification flags (vertex type, edge crossings, face crossings) }; private: - DataStructure& m_DataStructure; - const SurfaceNetsInputValues* m_InputValues = nullptr; - const std::atomic_bool& m_ShouldCancel; - const IFilter::MessageHandler& m_MessageHandler; + DataStructure& m_DataStructure; ///< Reference to the active DataStructure + const SurfaceNetsInputValues* m_InputValues = nullptr; ///< User parameters and created array paths + const std::atomic_bool& m_ShouldCancel; ///< User cancellation flag + const IFilter::MessageHandler& m_MessageHandler; ///< Progress message callback - // O(surface) data structures — populated during cell classification + /// O(surface) vertex storage -- populated during cell classification (Phase 1). + /// Each entry corresponds to one surface cell that will become a mesh vertex. std::vector m_Vertices; + + /// Hash map from packed padded-cell coordinate to index in m_Vertices. + /// Used for O(1) neighbor lookups during smoothing and edge-quad generation. std::unordered_map m_CellToVertex; }; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp index 521148a2f3..9c89c0e66b 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/TupleTransfer.hpp @@ -11,25 +11,65 @@ namespace nx::core { +/** + * @brief Record holding all data needed to perform one QuickSurfaceMesh face transfer. + * + * Batching many of these records into a single quickSurfaceTransferBatch() call + * allows the OOC-optimized implementation to compute the bounding range of source + * cell indices and face indices, then perform just two bulk I/O operations + * (one copyIntoBuffer for the source range, one copyFromBuffer for the destination + * range) instead of 2*N per-element accesses. + */ struct QuickSurfaceTransferData { - usize faceIndex = 0; - usize firstcIndex = 0; - usize secondcIndex = 0; - int32 faceLabel0 = 0; - int32 faceLabel1 = 0; + usize faceIndex = 0; ///< Index of the triangle face in the destination (face-level) array. + usize firstcIndex = 0; ///< Cell index on side 0 of the face (used when faceLabel0 != -1). + usize secondcIndex = 0; ///< Cell index on side 1 of the face (used when faceLabel1 != -1). + int32 faceLabel0 = 0; ///< Face label for side 0; -1 indicates exterior (skip copy). + int32 faceLabel1 = 0; ///< Face label for side 1; -1 indicates exterior (skip copy). }; +/** + * @brief Record holding all data needed to perform one SurfaceNets face transfer. + * + * Similar to QuickSurfaceTransferData but uses the SurfaceNets convention where + * each face has two associated NX-array indices (or max sentinel if exterior). + */ struct SurfaceNetsTransferData { - usize faceIndex = 0; + usize faceIndex = 0; ///< Index of the quad face in the destination (face-level) array. + /// Pair of NX-array cell indices for the two sides of the quad face. + /// A value of std::numeric_limits::max() indicates an exterior face (skip). std::array quadNxArrayIndices = {std::numeric_limits::max(), std::numeric_limits::max()}; }; /** - * @brief This is the base class that is used to transfer cell data to triangle face data - * but could be used generally to copy the tuple value from one Data Array to another - * DataArray of the same type. + * @brief Abstract base class for transferring tuple data from cell-level DataArrays + * to face-level DataArrays during surface mesh generation. + * + * @section overview Overview + * When QuickSurfaceMesh or SurfaceNets generates a triangle/quad mesh from a + * voxelized volume, each face sits between two cells. The face-level output + * arrays need to store the data from both adjacent cells (stored interleaved: + * [side0_comp0, side0_comp1, ..., side1_comp0, side1_comp1, ...]). + * + * @section ooc_optimization OOC Optimization: Batch Transfer Methods + * The original per-element transfer methods (quickSurfaceTransfer, surfaceNetsTransfer) + * use operator[] on the source and destination DataStore references. When the DataStore + * is backed by OOC chunked storage, each operator[] call may trigger a chunk load/evict + * cycle, making mesh generation extremely slow on large datasets. + * + * The batch methods (quickSurfaceTransferBatch, surfaceNetsTransferBatch) solve this by: + * 1. Scanning all records to find the bounding range of source cell indices and + * destination face indices. + * 2. Performing a single copyIntoBuffer() to bulk-read the source range into a + * local heap buffer. + * 3. Performing all tuple copies in-memory against the local buffers. + * 4. Performing a single copyFromBuffer() to bulk-write the destination range back. + * + * This reduces I/O from O(N) random accesses to O(1) sequential bulk operations per batch. + * Callers (e.g., QuickSurfaceMesh) accumulate records for a batch of faces and flush + * them periodically (every ~65K faces) to bound memory usage. */ class SIMPLNXCORE_EXPORT AbstractTupleTransfer { @@ -42,32 +82,67 @@ class SIMPLNXCORE_EXPORT AbstractTupleTransfer AbstractTupleTransfer& operator=(AbstractTupleTransfer&&) noexcept = delete; /** - * @brief - * @param faceIndex - * @param firstcIndex + * @brief Transfers one tuple from a source cell to a destination face (point sampling). + * + * Used by PointSampleTriangleGeom. Copies m_NumComps values from cellRef[firstcIndex...] + * to faceRef[faceIndex...]. + * + * @param faceIndex Starting value index in the destination face array. + * @param firstcIndex Starting value index in the source cell array. */ virtual void pointSampleTransfer(size_t faceIndex, size_t firstcIndex) = 0; /** - * @brief - * @param faceIndex - * @param firstcIndex - * @param secondcIndex - * @param faceLabels + * @brief Transfers cell data to both sides of a triangle face (per-element, non-batched). + * + * Copies cell data for side 0 and side 1 of a face, checking faceLabels to skip + * exterior faces (label == -1). The destination layout is interleaved: + * [face * numComps * 2 + 0..numComps-1] = side 0, [+ numComps..2*numComps-1] = side 1. + * + * @note This method uses per-element operator[] access. For OOC data, prefer + * accumulating QuickSurfaceTransferData records and calling quickSurfaceTransferBatch(). + * + * @param faceIndex Index of the face. + * @param firstcIndex Cell index for side 0 of the face. + * @param secondcIndex Cell index for side 1 of the face. + * @param faceLabels FaceLabels array; exterior faces have label -1. */ virtual void quickSurfaceTransfer(size_t faceIndex, size_t firstcIndex, size_t secondcIndex, AbstractDataStore& faceLabels) = 0; /** - * @brief - * @param faceIndex - * @param quadNxArrayIndices + * @brief Transfers cell data to both sides of a quad face for SurfaceNets (per-element). + * + * Same interleaved layout as quickSurfaceTransfer. Exterior sides are indicated + * by quadNxArrayIndices[i] == std::numeric_limits::max(). + * + * @note For OOC data, prefer accumulating SurfaceNetsTransferData records and + * calling surfaceNetsTransferBatch(). + * + * @param faceIndex Index of the quad face. + * @param quadNxArrayIndices Cell indices for the two sides of the quad; max sentinel = exterior. */ virtual void surfaceNetsTransfer(size_t faceIndex, const std::array& quadNxArrayIndices) = 0; + /** + * @brief OOC-optimized batch transfer for QuickSurfaceMesh faces. + * + * Processes a span of QuickSurfaceTransferData records in one bulk I/O round-trip. + * Default implementation is a no-op; subclasses override with typed bulk copy logic. + * + * @param records Span of transfer records to process in one batch. + */ virtual void quickSurfaceTransferBatch(nonstd::span /*records*/) { } + /** + * @brief OOC-optimized batch transfer for SurfaceNets faces. + * + * Processes a span of SurfaceNetsTransferData records in one bulk I/O round-trip. + * Default implementation is a no-op; subclasses override with typed bulk copy logic. + * + * @param records Span of transfer records to process in one batch. + */ virtual void surfaceNetsTransferBatch(nonstd::span /*records*/) { } @@ -75,11 +150,29 @@ class SIMPLNXCORE_EXPORT AbstractTupleTransfer protected: AbstractTupleTransfer() = default; - DataPath m_SourceDataPath; - DataPath m_DestinationDataPath; - size_t m_NumComps = 0; + DataPath m_SourceDataPath; ///< Path to the source (cell-level) DataArray in the DataStructure. + DataPath m_DestinationDataPath; ///< Path to the destination (face-level) DataArray in the DataStructure. + size_t m_NumComps = 0; ///< Number of components per tuple in both source and destination arrays. }; +/** + * @brief Typed implementation of AbstractTupleTransfer for direct cell-to-face data transfer. + * + * Copies data directly from cell-level DataArrays to face-level DataArrays. + * The source array is indexed by cell index and the destination array stores two + * sides per face in interleaved layout. + * + * @section ooc_batch OOC Batch Methods + * The quickSurfaceTransferBatch() and surfaceNetsTransferBatch() overrides implement + * the bulk I/O strategy described in AbstractTupleTransfer: they scan the batch of + * records to find the [minSrc, maxSrc] cell index range and [minFace, maxFace] face + * index range, perform a single copyIntoBuffer to read the source cell data, execute + * all tuple copies in-memory, and perform a single copyFromBuffer to write the + * destination face data. This converts O(N) random OOC accesses per batch into + * exactly 2 bulk I/O operations. + * + * @tparam T The element type of both the source cell DataArray and destination face DataArray. + */ template class TransferTuple : public AbstractTupleTransfer { @@ -88,10 +181,10 @@ class TransferTuple : public AbstractTupleTransfer using DataStoreType = AbstractDataStore; /** - * @brief - * @param dataStructure Current DataStructure - * @param selectedDataPath The source data path - * @param createdArrayPath The destination data path + * @brief Constructs a TransferTuple for direct cell-to-face data transfer. + * @param dataStructure Current DataStructure containing both arrays. + * @param selectedDataPath Path to the source (cell-level) DataArray. + * @param createdArrayPath Path to the destination (face-level) DataArray. */ TransferTuple(DataStructure& dataStructure, const DataPath& selectedDataPath, const DataPath& createdArrayPath) : m_CellRef(dataStructure.template getDataRefAs(selectedDataPath).getDataStoreRef()) @@ -177,6 +270,18 @@ class TransferTuple : public AbstractTupleTransfer } } + /** + * @brief OOC-optimized batch transfer for QuickSurfaceMesh. + * + * Replaces N per-element operator[] calls with 2 bulk I/O operations: + * - 1x copyIntoBuffer: reads contiguous source cell range [minSrc..maxSrc] + * - 1x copyFromBuffer: writes contiguous destination face range [minFace..maxFace] + * + * All tuple copies happen in-memory between heap-allocated local buffers. + * Exterior faces (faceLabel == -1) are skipped during the copy phase. + * + * @param records Span of QuickSurfaceTransferData records for this batch. + */ void quickSurfaceTransferBatch(nonstd::span records) override { if(records.empty()) @@ -184,7 +289,8 @@ class TransferTuple : public AbstractTupleTransfer return; } - // Find source cell index range and face index range + // Phase 1: Scan records to find the bounding range of source cell indices and face indices. + // This determines the minimum contiguous region we need to bulk-read/write. usize minSrc = std::numeric_limits::max(); usize maxSrc = 0; usize minFace = std::numeric_limits::max(); @@ -207,20 +313,21 @@ class TransferTuple : public AbstractTupleTransfer if(minSrc > maxSrc) { - return; // all exterior faces, nothing to copy from source + return; // All exterior faces in this batch; nothing to copy from source } - // Bulk read source cell data + // Phase 2: Bulk-read the source cell data range into a local buffer. + // This is the key OOC optimization -- one I/O call instead of N element reads. usize srcTupleCount = maxSrc - minSrc + 1; auto srcBuf = std::make_unique(srcTupleCount * m_NumComps); m_CellRef.copyIntoBuffer(minSrc * m_NumComps, nonstd::span(srcBuf.get(), srcTupleCount * m_NumComps)); - // Build destination buffer + // Allocate destination buffer for the face range (2 sides per face, each with m_NumComps) usize faceCount = maxFace - minFace + 1; auto destBuf = std::make_unique(faceCount * m_NumComps * 2); std::fill_n(destBuf.get(), faceCount * m_NumComps * 2, T{}); - // Process all records using local buffers + // Phase 3: Process all records using local buffers (pure in-memory copies) for(const auto& r : records) { usize localFace = r.faceIndex - minFace; @@ -244,11 +351,18 @@ class TransferTuple : public AbstractTupleTransfer } } - // Bulk write destination face data + // Phase 4: Bulk-write the destination face data back. m_FaceRef.copyFromBuffer(minFace * m_NumComps * 2, nonstd::span(destBuf.get(), faceCount * m_NumComps * 2)); } - + /** + * @brief OOC-optimized batch transfer for SurfaceNets. + * + * Same bulk I/O strategy as quickSurfaceTransferBatch but using SurfaceNets + * conventions: exterior sides are indicated by quadNxArrayIndices[i] == max sentinel. + * + * @param records Span of SurfaceNetsTransferData records for this batch. + */ void surfaceNetsTransferBatch(nonstd::span records) override { if(records.empty()) @@ -318,10 +432,35 @@ class TransferTuple : public AbstractTupleTransfer } private: - DataStoreType& m_CellRef; - DataStoreType& m_FaceRef; + DataStoreType& m_CellRef; ///< Reference to the source (cell-level) DataStore. + DataStoreType& m_FaceRef; ///< Reference to the destination (face-level) DataStore. }; +/** + * @brief Typed implementation of AbstractTupleTransfer for feature-level to face-level transfer. + * + * Unlike TransferTuple which copies cell data directly, this class performs an indirection + * through a FeatureIds array: cell index -> featureId -> feature-level data -> face output. + * This is used when surface mesh faces need feature-level attributes (e.g., average + * orientations, average C-axis values) rather than raw cell-level data. + * + * @section ooc_batch OOC Batch Methods + * The batch methods follow the same 4-phase pattern as TransferTuple but with an + * additional data source: + * - Phase 1: Scan records for cell index range and face index range. + * - Phase 2: Bulk-read the FeatureIds for the cell range (1 copyIntoBuffer call). + * - Phase 3: Bulk-read the entire feature-level data array (typically small -- tens + * of thousands of features vs. millions of cells). This is cached entirely since + * feature IDs can map to any feature. + * - Phase 4: Process all records in-memory: look up featureId from the cell buffer, + * then copy the feature data into the face destination buffer. + * - Phase 5: Bulk-write the destination face data (1 copyFromBuffer call). + * + * Total OOC I/O: 3 bulk operations instead of 3*N per-element accesses. + * + * @tparam T The element type of the feature-level source and face-level destination DataArrays. + * @tparam K The element type of the FeatureIds array (typically int32). + */ template class TransferFeatureTuple : public AbstractTupleTransfer { @@ -332,10 +471,11 @@ class TransferFeatureTuple : public AbstractTupleTransfer using FeatureIdsStoreType = AbstractDataStore; /** - * @brief - * @param dataStructure Current DataStructure - * @param selectedDataPath The source data path - * @param createdArrayPath The destination data path + * @brief Constructs a TransferFeatureTuple for feature-level to face-level transfer. + * @param dataStructure Current DataStructure containing all arrays. + * @param selectedDataPath Path to the source (feature-level) DataArray. + * @param createdArrayPath Path to the destination (face-level) DataArray. + * @param featureIdsArrayPath Path to the FeatureIds array that maps cell index -> feature ID. */ TransferFeatureTuple(DataStructure& dataStructure, const DataPath& selectedDataPath, const DataPath& createdArrayPath, const DataPath& featureIdsArrayPath) : m_FeatureDataRef(dataStructure.template getDataRefAs(selectedDataPath).getDataStoreRef()) @@ -432,6 +572,21 @@ class TransferFeatureTuple : public AbstractTupleTransfer } } + /** + * @brief OOC-optimized batch transfer for QuickSurfaceMesh (feature-level variant). + * + * Unlike the TransferTuple version, this performs a two-level indirection: + * 1. Bulk-read FeatureIds for the cell range (cell index -> feature ID). + * 2. Bulk-read the entire feature-level data array (small, typically thousands of features). + * 3. For each record: look up the feature ID, copy feature data into the face buffer. + * 4. Bulk-write the face buffer. + * + * The feature data array is read in its entirety because feature IDs can be arbitrary + * (not necessarily contiguous with the cell indices), and the array is small enough + * that caching it all is more efficient than computing the feature ID range. + * + * @param records Span of QuickSurfaceTransferData records for this batch. + */ void quickSurfaceTransferBatch(nonstd::span records) override { if(records.empty()) @@ -439,7 +594,7 @@ class TransferFeatureTuple : public AbstractTupleTransfer return; } - // Find cell index range and face index range + // Phase 1: Scan records for bounding cell index range and face index range usize minSrc = std::numeric_limits::max(); usize maxSrc = 0; usize minFace = std::numeric_limits::max(); @@ -465,22 +620,24 @@ class TransferFeatureTuple : public AbstractTupleTransfer return; } - // Bulk read featureIds for the cell range + // Phase 2: Bulk-read FeatureIds for the cell range (cell-level, contiguous range) usize srcTupleCount = maxSrc - minSrc + 1; auto featureIdBuf = std::make_unique(srcTupleCount); m_FeatureIdsRef.copyIntoBuffer(minSrc, nonstd::span(featureIdBuf.get(), srcTupleCount)); - // Feature data is small (feature-level, not cell-level) — cache it all + // Phase 3: Cache the entire feature-level data array locally. + // Feature arrays are small (one tuple per grain, typically thousands) so reading + // the full array is both simpler and faster than computing the needed feature ID range. usize featureTuples = m_FeatureDataRef.getNumberOfTuples(); auto featureDataBuf = std::make_unique(featureTuples * m_NumComps); m_FeatureDataRef.copyIntoBuffer(0, nonstd::span(featureDataBuf.get(), featureTuples * m_NumComps)); - // Build destination buffer + // Allocate destination face buffer usize faceCount = maxFace - minFace + 1; auto destBuf = std::make_unique(faceCount * m_NumComps * 2); std::fill_n(destBuf.get(), faceCount * m_NumComps * 2, T{}); - // Process records + // Phase 4: Process records in-memory: cell index -> featureId -> feature data -> face buffer for(const auto& r : records) { usize localFace = r.faceIndex - minFace; @@ -506,11 +663,18 @@ class TransferFeatureTuple : public AbstractTupleTransfer } } - // Bulk write destination + // Phase 5: Bulk-write the destination face data m_FaceRef.copyFromBuffer(minFace * m_NumComps * 2, nonstd::span(destBuf.get(), faceCount * m_NumComps * 2)); } - + /** + * @brief OOC-optimized batch transfer for SurfaceNets (feature-level variant). + * + * Same two-level indirection as quickSurfaceTransferBatch (cell -> featureId -> feature data) + * but using SurfaceNets conventions for exterior-face detection. + * + * @param records Span of SurfaceNetsTransferData records for this batch. + */ void surfaceNetsTransferBatch(nonstd::span records) override { if(records.empty()) @@ -586,28 +750,39 @@ class TransferFeatureTuple : public AbstractTupleTransfer } private: - DataStoreType& m_FeatureDataRef; - DataStoreType& m_FaceRef; - FeatureIdsStoreType& m_FeatureIdsRef; + DataStoreType& m_FeatureDataRef; ///< Reference to the source (feature-level) DataStore. + DataStoreType& m_FaceRef; ///< Reference to the destination (face-level) DataStore. + FeatureIdsStoreType& m_FeatureIdsRef; ///< Reference to the FeatureIds DataStore (cell index -> feature ID). }; /** + * @brief Factory function that creates a type-appropriate TransferTuple instance and appends it + * to the provided vector of tuple transfer functions. + * + * Inspects the DataType of the selected DataArray and instantiates TransferTuple with + * the matching type. This is the primary way callers set up direct cell-to-face transfers. * - * @param dataStructure - * @param selectedDataPath - * @param createdDataPath - * @param tupleTransferFunctions + * @param dataStructure Current DataStructure. + * @param selectedDataPath Path to the source (cell-level) DataArray. + * @param createdDataPath Path to the destination (face-level) DataArray. + * @param tupleTransferFunctions Vector to append the new instance to. */ SIMPLNXCORE_EXPORT void AddTupleTransferInstance(DataStructure& dataStructure, const DataPath& selectedDataPath, const DataPath& createdDataPath, std::vector>& tupleTransferFunctions); /** + * @brief Factory function that creates a type-appropriate TransferFeatureTuple instance and + * appends it to the provided vector of tuple transfer functions. + * + * Inspects the DataType of the selected DataArray and the FeatureIds array, then + * instantiates TransferFeatureTuple with the matching types. This sets up + * the two-level indirection path: cell -> featureId -> feature data -> face output. * - * @param dataStructure - * @param selectedDataPath - * @param createdDataPath - * @param featureIdsArrayPath - * @param tupleTransferFunctions + * @param dataStructure Current DataStructure. + * @param selectedDataPath Path to the source (feature-level) DataArray. + * @param createdDataPath Path to the destination (face-level) DataArray. + * @param featureIdsArrayPath Path to the FeatureIds array (cell index -> feature ID). + * @param tupleTransferFunctions Vector to append the new instance to. */ SIMPLNXCORE_EXPORT void AddFeatureTupleTransferInstance(DataStructure& dataStructure, const DataPath& selectedDataPath, const DataPath& createdDataPath, const DataPath& featureIdsArrayPath, std::vector>& tupleTransferFunctions); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp index c3aef001fa..4e9369c313 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.cpp @@ -75,6 +75,23 @@ Result<> WriteAvizoRectilinearCoordinate::generateHeader(FILE* outputFile) const } // ----------------------------------------------------------------------------- +/** + * @brief Writes the FeatureIds and rectilinear coordinate data to the Avizo output file. + * + * @section ooc_strategy OOC Strategy + * The FeatureIds array can be very large (millions of voxels). The original implementation + * used featureIds.data() to get a raw pointer and fwrite the entire array, but this fails + * when the DataStore is out-of-core because data() is not available. + * + * The optimized version reads in chunks of k_ChunkSize (65536) tuples via copyIntoBuffer(), + * then writes each chunk to the output file. This: + * - Works with any DataStore backend (in-memory or OOC). + * - Bounds memory to ~256 KB (65536 * sizeof(int32)) regardless of volume size. + * - Maintains sequential I/O pattern for both the DataStore reads and file writes. + * + * @param outputFile FILE pointer to the open Avizo output file. + * @return Result<> indicating success. + */ Result<> WriteAvizoRectilinearCoordinate::writeData(FILE* outputFile) const { const auto& geom = m_DataStructure.getDataRefAs(m_InputValues->GeometryPath); @@ -87,6 +104,7 @@ Result<> WriteAvizoRectilinearCoordinate::writeData(FILE* outputFile) const const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); const usize totalPoints = featureIds.getNumberOfTuples(); + // Read FeatureIds in chunks via copyIntoBuffer() (OOC-safe) and write each chunk to file constexpr usize k_ChunkSize = 65536; std::vector buffer(k_ChunkSize); const auto& featureIdsStore = featureIds.getDataStoreRef(); @@ -106,7 +124,8 @@ Result<> WriteAvizoRectilinearCoordinate::writeData(FILE* outputFile) const } else { - // The "20 Items" is purely arbitrary and is put in to try and save some space in the ASCII file + // ASCII mode: read chunks, format each value individually. + // The "20 items per line" formatting is preserved from the original code. int itemCount = 0; for(usize offset = 0; offset < totalPoints; offset += k_ChunkSize) { diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.hpp index d21b928acd..26f5068707 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoRectilinearCoordinate.hpp @@ -7,9 +7,13 @@ namespace nx::core { /** * @class WriteAvizoRectilinearCoordinate - * @brief This filter writes out a native Avizo Rectilinear Coordinate data file. + * @brief Writes an Avizo Rectilinear Coordinate data file containing FeatureIds and axis coordinates. + * + * @section ooc_summary OOC Optimization Summary + * The FeatureIds array is read in chunks of 64K tuples via copyIntoBuffer() and written + * to the output file per chunk, replacing the original raw pointer fwrite that required + * in-memory data. This works with both in-memory and OOC DataStore backends. */ - class SIMPLNXCORE_EXPORT WriteAvizoRectilinearCoordinate : public AvizoWriter { public: @@ -21,10 +25,22 @@ class SIMPLNXCORE_EXPORT WriteAvizoRectilinearCoordinate : public AvizoWriter WriteAvizoRectilinearCoordinate& operator=(const WriteAvizoRectilinearCoordinate&) = delete; WriteAvizoRectilinearCoordinate& operator=(WriteAvizoRectilinearCoordinate&&) noexcept = delete; + /** + * @brief Executes the writer: generates header and writes data. + */ Result<> operator()(); protected: + /** + * @brief Generates the Avizo rectilinear coordinate file header. + * @param outputFile FILE pointer to the open output file. + */ Result<> generateHeader(FILE* outputFile) const override; + + /** + * @brief Writes FeatureIds and rectilinear coordinates using OOC-safe chunked I/O. + * @param outputFile FILE pointer to the open output file. + */ Result<> writeData(FILE* outputFile) const override; }; diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp index d5a64ddf2a..543a92804f 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp @@ -82,6 +82,18 @@ Result<> WriteAvizoUniformCoordinate::generateHeader(FILE* outputFile) const } // ----------------------------------------------------------------------------- +/** + * @brief Writes the FeatureIds data to the Avizo uniform coordinate output file. + * + * @section ooc_strategy OOC Strategy + * Same chunked copyIntoBuffer() approach as WriteAvizoRectilinearCoordinate::writeData(). + * The FeatureIds array is read in 64K-tuple chunks to avoid per-element OOC access, + * and each chunk is written to the output file in one fwrite (binary) or formatted + * fprintf loop (ASCII). + * + * @param outputFile FILE pointer to the open Avizo output file. + * @return Result<> indicating success. + */ Result<> WriteAvizoUniformCoordinate::writeData(FILE* outputFile) const { fprintf(outputFile, "@1\n"); @@ -89,6 +101,7 @@ Result<> WriteAvizoUniformCoordinate::writeData(FILE* outputFile) const const auto& featureIds = m_DataStructure.getDataRefAs(m_InputValues->FeatureIdsArrayPath); const usize totalPoints = featureIds.getNumberOfTuples(); + // Chunked OOC-safe read + file write pattern (see WriteAvizoRectilinearCoordinate for details) constexpr usize k_ChunkSize = 65536; std::vector buffer(k_ChunkSize); const auto& featureIdsStore = featureIds.getDataStoreRef(); diff --git a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.hpp b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.hpp index 79c1b55ff3..ff0819531c 100644 --- a/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.hpp +++ b/src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.hpp @@ -7,9 +7,12 @@ namespace nx::core { /** * @class WriteAvizoUniformCoordinate - * @brief This filter writes out a native Avizo Uniform Coordinate data file. + * @brief Writes an Avizo Uniform Coordinate data file containing FeatureIds. + * + * @section ooc_summary OOC Optimization Summary + * Same chunked copyIntoBuffer() approach as WriteAvizoRectilinearCoordinate. + * The FeatureIds array is read in 64K-tuple chunks to support OOC DataStore backends. */ - class SIMPLNXCORE_EXPORT WriteAvizoUniformCoordinate : public AvizoWriter { public: @@ -21,10 +24,22 @@ class SIMPLNXCORE_EXPORT WriteAvizoUniformCoordinate : public AvizoWriter WriteAvizoUniformCoordinate& operator=(const WriteAvizoUniformCoordinate&) = delete; WriteAvizoUniformCoordinate& operator=(WriteAvizoUniformCoordinate&&) noexcept = delete; + /** + * @brief Executes the writer: generates header and writes data. + */ Result<> operator()(); protected: + /** + * @brief Generates the Avizo uniform coordinate file header. + * @param outputFile FILE pointer to the open output file. + */ Result<> generateHeader(FILE* outputFile) const override; + + /** + * @brief Writes FeatureIds using OOC-safe chunked I/O. + * @param outputFile FILE pointer to the open output file. + */ Result<> writeData(FILE* outputFile) const override; }; diff --git a/src/simplnx/Utilities/SliceBufferedTransfer.hpp b/src/simplnx/Utilities/SliceBufferedTransfer.hpp index 737e5e2208..77c2416e67 100644 --- a/src/simplnx/Utilities/SliceBufferedTransfer.hpp +++ b/src/simplnx/Utilities/SliceBufferedTransfer.hpp @@ -15,44 +15,111 @@ namespace nx::core { /** - * @brief Performs Z-slice buffered tuple transfer for a single typed DataArray. + * @brief Type-dispatched functor that performs Z-slice buffered tuple transfer for a single DataArray. * - * Reads source and destination Z-slices into memory, copies tuples in-memory - * based on a neighbors mapping, then writes back. This eliminates per-element - * OOC store access during transfer phases. + * @section ooc_motivation OOC Motivation + * Morphological fill algorithms (FillBadData, ErodeDilateCoordinationNumber, etc.) use a + * "neighbors" mapping where each voxel stores the global index of the voxel it should copy + * data from. When the underlying DataStore is out-of-core (OOC), accessing elements via + * operator[] triggers chunk load/evict cycles -- one for the source read AND one for the + * destination write -- for every single voxel. On large datasets this makes the algorithm + * 100-1000x slower than in-memory. * - * Requires that source indices are always within ±1 Z-slice of the destination - * (true for face-neighbor morphological algorithms). + * @section approach Approach: 3-Slot Rolling Window + * Because face-neighbor morphological algorithms only ever copy from a source voxel that + * is within +/-1 Z-slice of the destination, we can exploit this locality: + * + * 1. Maintain a rolling window of 3 source-slice buffers: slot 0 = z-1, slot 1 = z, slot 2 = z+1. + * 2. For each Z-slice, bulk-read the destination slice into a local buffer. + * 3. For each voxel in the slice that needs copying, look up the source from the + * appropriate slot (already in memory) and copy into the destination buffer. + * 4. Bulk-write the modified destination slice back. + * 5. Advance the rolling window: swap slots, read the next z+1 slice. + * + * This reduces OOC I/O from O(N) random element accesses to O(dimZ) sequential bulk reads/writes, + * where each bulk operation transfers an entire slice in one copyIntoBuffer/copyFromBuffer call. + * + * @section memory Memory Footprint + * 4 slices worth of element data: 3 source slots + 1 destination buffer. + * For a 1024x1024 float32 array, that is approximately 16 MB -- trivial compared to + * the full volume which could be many GB. + * + * @section bool_handling bool Specialization + * std::vector is bit-packed and cannot provide a contiguous T* pointer, so + * BufferType uses std::unique_ptr for bool and std::vector for everything else. */ struct SliceBufferedTransferFunctor { + /** + * @brief Buffer type alias that avoids std::vector bit-packing. + * + * std::vector is specialized to use 1 bit per element, which means + * data() returns a proxy, not a bool*. The copyIntoBuffer/copyFromBuffer + * API requires a contiguous T* span, so we use unique_ptr instead. + * + * @tparam T The element type of the DataArray being transferred. + */ template using BufferType = std::conditional_t, std::unique_ptr, std::vector>; + /** + * @brief Returns a raw pointer to the underlying buffer data (vector overload). + * @tparam T Element type. + * @param v The vector buffer. + * @return Pointer to the first element. + */ template static T* bufPtr(std::vector& v) { return v.data(); } + /** + * @brief Returns a raw pointer to the underlying buffer data (unique_ptr overload). + * @tparam T Element type. + * @param p The unique_ptr buffer (used for bool specialization). + * @return Pointer to the first element. + */ template static T* bufPtr(std::unique_ptr& p) { return p.get(); } + /** + * @brief Returns a const raw pointer to the underlying buffer data (const vector overload). + * @tparam T Element type. + * @param v The vector buffer. + * @return Const pointer to the first element. + */ template static const T* bufPtr(const std::vector& v) { return v.data(); } + /** + * @brief Returns a const raw pointer to the underlying buffer data (const unique_ptr overload). + * @tparam T Element type. + * @param p The unique_ptr buffer (used for bool specialization). + * @return Const pointer to the first element. + */ template static const T* bufPtr(const std::unique_ptr& p) { return p.get(); } + /** + * @brief Factory method to create a buffer of the appropriate type for T. + * + * For bool, allocates a unique_ptr to avoid std::vector bit-packing. + * For all other types, allocates a std::vector. + * + * @tparam T Element type. + * @param size Number of elements to allocate. + * @return A BufferType with the requested capacity. + */ template static BufferType makeBuf(usize size) { @@ -66,14 +133,34 @@ struct SliceBufferedTransferFunctor } } + /** + * @brief Performs the Z-slice buffered transfer for a typed DataArray. + * + * Iterates through all Z-slices, maintaining a 3-slot rolling source window. + * For each destination voxel where neighbors[destIdx] >= 0 and shouldCopy(destIdx) + * is true, copies the full tuple (all components) from the source location into + * the destination buffer. Modified destination slices are written back via + * copyFromBuffer. + * + * @tparam T The element type of the DataArray (dispatched by ExecuteDataFunction). + * @param dataArray The DataArray to transfer data within (modified in place). + * @param neighbors Global neighbor mapping: neighbors[i] is the global linear index + * of the source voxel for destination voxel i, or -1 if no copy is needed. + * @param sliceSize Number of voxels per Z-slice (dimX * dimY). + * @param dimZ Number of Z-slices in the volume. + * @param shouldCopy Predicate returning true if the voxel at the given global index + * should be overwritten. Allows callers to skip voxels that are already valid. + */ template void operator()(IDataArray& dataArray, const std::vector& neighbors, usize sliceSize, usize dimZ, const std::function& shouldCopy) { auto& store = dynamic_cast&>(dataArray).getDataStoreRef(); const usize numComp = store.getNumberOfComponents(); + // sliceValues = number of T elements per Z-slice (voxels * components) const usize sliceValues = sliceSize * numComp; - // Rolling source window: slot 0=z-1, slot 1=z, slot 2=z+1 + // Rolling source window: slot 0 = z-1, slot 1 = z, slot 2 = z+1 + // These three buffers are reused across all Z iterations via std::swap std::array, 3> srcSlices; for(auto& s : srcSlices) { @@ -81,9 +168,10 @@ struct SliceBufferedTransferFunctor } auto destSlice = makeBuf(sliceValues); + // Lambda to bulk-read one Z-slice from the OOC store into a specific slot auto readSlice = [&](usize z, usize slot) { store.copyIntoBuffer(z * sliceValues, nonstd::span(bufPtr(srcSlices[slot]), sliceValues)); }; - // Initialize source rolling window + // Prime the rolling window: read z=0 into slot 1 (current), z=1 into slot 2 (next) readSlice(0, 1); if(dimZ > 1) { @@ -92,17 +180,18 @@ struct SliceBufferedTransferFunctor for(usize zIdx = 0; zIdx < dimZ; zIdx++) { + // Advance the rolling window by shifting slot contents down if(zIdx > 0) { - std::swap(srcSlices[0], srcSlices[1]); - std::swap(srcSlices[1], srcSlices[2]); + std::swap(srcSlices[0], srcSlices[1]); // Old "current" becomes "previous" + std::swap(srcSlices[1], srcSlices[2]); // Old "next" becomes "current" if(zIdx + 1 < dimZ) { - readSlice(zIdx + 1, 2); + readSlice(zIdx + 1, 2); // Read the new "next" slice } } - // Read dest slice + // Bulk-read the destination slice so we can modify it in memory store.copyIntoBuffer(zIdx * sliceValues, nonstd::span(bufPtr(destSlice), sliceValues)); bool modified = false; @@ -112,18 +201,20 @@ struct SliceBufferedTransferFunctor const int64 srcIdx = neighbors[destIdx]; if(srcIdx >= 0 && shouldCopy(destIdx)) { + // Determine which rolling-window slot contains the source data const usize srcZ = static_cast(srcIdx) / sliceSize; const usize srcInSlice = static_cast(srcIdx) % sliceSize; - usize srcSlot = 1; + usize srcSlot = 1; // Same Z-slice (most common case) if(srcZ < zIdx) { - srcSlot = 0; + srcSlot = 0; // Source is in previous Z-slice } else if(srcZ > zIdx) { - srcSlot = 2; + srcSlot = 2; // Source is in next Z-slice } + // Copy all components of the source tuple into the destination buffer for(usize c = 0; c < numComp; c++) { bufPtr(destSlice)[inSlice * numComp + c] = bufPtr(srcSlices[srcSlot])[srcInSlice * numComp + c]; @@ -132,6 +223,7 @@ struct SliceBufferedTransferFunctor } } + // Only write back if at least one tuple was modified (avoids unnecessary OOC writes) if(modified) { store.copyFromBuffer(zIdx * sliceValues, nonstd::span(bufPtr(destSlice), sliceValues)); @@ -142,7 +234,16 @@ struct SliceBufferedTransferFunctor /** * @brief Convenience function to perform slice-buffered transfer on a single IDataArray. - * Dispatches on the array's DataType to call the typed implementation. + * + * Dispatches on the array's DataType via ExecuteDataFunction to invoke the typed + * SliceBufferedTransferFunctor::operator(). This is the primary entry point for + * algorithms that have a full-volume neighbors mapping and want OOC-safe data transfer. + * + * @param dataArray The DataArray to transfer data within (modified in place). + * @param neighbors Global neighbor mapping: neighbors[i] = source index for voxel i, or -1. + * @param sliceSize Number of voxels per Z-slice (dimX * dimY). + * @param dimZ Number of Z-slices. + * @param shouldCopy Predicate: return true if the voxel at the given index should be overwritten. */ inline void SliceBufferedTransfer(IDataArray& dataArray, const std::vector& neighbors, usize sliceSize, usize dimZ, const std::function& shouldCopy) { @@ -150,17 +251,38 @@ inline void SliceBufferedTransfer(IDataArray& dataArray, const std::vector= 0, copies the tuple - * from the source location (which must be within ±1 Z of destZ). Manages its own - * 3-slice source rolling window internally. + * @section approach Approach: On-Demand Lazy Loading + * For each voxel in the destination Z-slice where sliceMarks[inSlice] >= 0: + * 1. Compute which Z-slice the source voxel is on (must be within +/-1 of destZ). + * 2. Lazily load that source Z-slice if not already in memory. + * 3. Copy the tuple from the source buffer into the destination buffer. + * 4. Bulk-write the destination slice back when done. * - * This is the building block for O(sliceSize) memory algorithms that eliminate - * full-volume neighbor arrays. + * @section memory Memory Footprint + * At most 4 slices: 1 destination + up to 3 source (z-1, z, z+1), each loaded on demand. */ struct SliceTransferOneZFunctor { + /** + * @brief Transfers tuples for a single Z-slice based on per-slice marks. + * + * @tparam T The element type of the DataArray (dispatched by ExecuteDataFunction). + * @param dataArray The DataArray to transfer data within (modified in place). + * @param sliceMarks Per-slice marks of size sliceSize. sliceMarks[inSlice] is the global + * linear index of the source voxel, or -1 if no copy is needed for this position. + * @param sliceSize Number of voxels per Z-slice (dimX * dimY). + * @param destZ The Z-index of the destination slice being processed. + * @param dimZ Total number of Z-slices (used for bounds checking). + */ template void operator()(IDataArray& dataArray, const std::vector& sliceMarks, usize sliceSize, usize destZ, usize dimZ) { @@ -169,14 +291,15 @@ struct SliceTransferOneZFunctor const usize numComp = store.getNumberOfComponents(); const usize sliceValues = sliceSize * numComp; - // Read dest slice + // Bulk-read the destination slice into a local buffer auto destBuf = SliceBufferedTransferFunctor::makeBuf(sliceValues); store.copyIntoBuffer(destZ * sliceValues, nonstd::span(SliceBufferedTransferFunctor::bufPtr(destBuf), sliceValues)); - // Read source slices on demand (only those needed) - std::array srcBufs; // 0=z-1, 1=z, 2=z+1 + // Source slice buffers loaded lazily on demand: slot 0 = z-1, slot 1 = z, slot 2 = z+1 + std::array srcBufs; std::array srcLoaded = {false, false, false}; + // Lazy-load helper: only reads a source Z-slice the first time it is needed auto ensureSrcLoaded = [&](usize slot, usize srcZ) { if(!srcLoaded[slot] && srcZ < dimZ) { @@ -192,12 +315,14 @@ struct SliceTransferOneZFunctor const int64 srcGlobalIdx = sliceMarks[inSlice]; if(srcGlobalIdx < 0) { - continue; + continue; // No copy needed for this voxel } + // Determine which Z-slice the source voxel is on and its in-slice offset const usize srcZ = static_cast(srcGlobalIdx) / sliceSize; const usize srcInSlice = static_cast(srcGlobalIdx) % sliceSize; + // Map source Z to rolling-window slot: 0 = previous, 1 = same, 2 = next usize srcSlot = 1; if(srcZ < destZ) { @@ -208,8 +333,10 @@ struct SliceTransferOneZFunctor srcSlot = 2; } + // Ensure the source slice is loaded before reading from it ensureSrcLoaded(srcSlot, srcZ); + // Copy all components of the source tuple into the destination buffer for(usize c = 0; c < numComp; c++) { SliceBufferedTransferFunctor::bufPtr(destBuf)[inSlice * numComp + c] = SliceBufferedTransferFunctor::bufPtr(srcBufs[srcSlot])[srcInSlice * numComp + c]; @@ -217,6 +344,7 @@ struct SliceTransferOneZFunctor modified = true; } + // Only write back if at least one tuple was modified if(modified) { store.copyFromBuffer(destZ * sliceValues, nonstd::span(SliceBufferedTransferFunctor::bufPtr(destBuf), sliceValues)); @@ -225,7 +353,17 @@ struct SliceTransferOneZFunctor }; /** - * @brief Convenience function to transfer a single Z-slice using per-slice marks. + * @brief Convenience function to transfer a single Z-slice of an IDataArray using per-slice marks. + * + * Dispatches on the array's DataType via ExecuteDataFunction to invoke the typed + * SliceTransferOneZFunctor::operator(). This is the entry point for algorithms that + * process one Z-slice at a time and maintain a per-slice marks array. + * + * @param dataArray The DataArray to transfer data within (modified in place). + * @param sliceMarks Per-slice marks: sliceMarks[inSlice] = global source index, or -1. + * @param sliceSize Number of voxels per Z-slice (dimX * dimY). + * @param destZ The Z-index of the destination slice being processed. + * @param dimZ Total number of Z-slices (for bounds checking). */ inline void SliceBufferedTransferOneZ(IDataArray& dataArray, const std::vector& sliceMarks, usize sliceSize, usize destZ, usize dimZ) {