S390x test fixes

[AlekseiNikiforovIBM]

Description

This PR contains fixes to various big endian support issues in onnxruntime, both in libraries and tests.

Motivation and Context

Currently some tests from onnxruntime testsuite fail.
This change fixes all tests from onnxruntime testsuite when it's built without training support.
It also includes a linking issue fix.

Following tests are fixed on s390x:
OrtModelOnlyTests.ValidateOrtFormatModelDoesNotRunOptimizersInFullBuild
FlatbufferUtilsTest.ExternalWriteReadWithLoadInitializers
SparseTensorConversionTests.SparseTensorProtoToDense_Rank1Indices64
SparseTensorConversionTests.SparseTensorProtoToDense_Rank1Indices32
SparseTensorConversionTests.SparseTensorProtoToDense_Rank1Indices16
SparseTensorConversionTests.SparseTensorProtoToDense_Rank1Indices8
SparseTensorConversionTests.SparseTensorProtoToDense_Rank2Indices_COO
SparseTensorConversionTests.TestConstantNodeConversion
OrtModelOnlyTests.SparseInitializerHandling
SparseTensorConversionTests.TestConstantNodeConversion
SparseTensorConversionTests.TestDenseToSparseConversion
ExecutionFrameTestInit.SparseInitializerAsOutput
CApiTest.SparseOutputModel

[AlekseiNikiforovIBM]

I've added fixes for tests enabled with training. Although tests are for training, a lot of fixes are actually in common code.

[AlekseiNikiforovIBM]

@tianleiwu @amarin16 @baijumeswani, could you please take a look?

[tianleiwu]

1. onnxruntime/core/graph/graph_flatbuffers_utils.cc (Changes Requested)

In SaveInitializerOrtFormat, the handling of HasExternalData(be_copy) for big-endian machines introduces a bug where external data will be saved into the ORT Flatbuffer as Big-Endian bytes instead of Little-Endian.

Concrete byte-level trace of the bug:

For the inline data path (correct), using int32 value 1 as an example:

1. be_copy.raw_data() starts as LE bytes from the ONNX proto: [01, 00, 00, 00]
2. ConvertRawDataInTensorProto(be_copy) swaps in-place → [00, 00, 00, 01] (now BE)
3. UnpackInitializerData → has raw_data → UnpackTensorWithRawData → ReadLittleEndian:
   interprets [00, 00, 00, 01] as LE, swaps to native BE → output = [01, 00, 00, 00]
4. Result: unpacked_tensor = [01, 00, 00, 00] = LE bytes ✅

For the external data path (buggy):

1. ConvertRawDataInTensorProto(be_copy) — no-op (external data, no inline data to swap)
2. TensorProtoWithExternalDataToTensorProto(be_copy_external_data, {}, be_copy) — loads LE bytes from disk into a brand new TensorProto result (see tensorprotoutils.cc:296), which only copies name/data_type/dims but NOT external_data or data_location. So be_copy becomes an inline-data TensorProto with raw_data = [01, 00, 00, 00] (LE from disk).
3. UnpackInitializerData(be_copy) — HasExternalData now returns false, so it takes the non-external path → CASE_UNPACK → UnpackTensorWithRawData → ReadLittleEndian:
   interprets [01, 00, 00, 00] as LE, swaps to native BE → output = [00, 00, 00, 01]
4. Result: unpacked_tensor = [00, 00, 00, 01] = BE bytes ❌ — flatbuffer stores BE data.

Root cause: The inline data path relies on a double-swap (step 2 + step 3) that cancels out, producing LE. The external data path only has a single swap (step 3), producing BE.

Proposed fix — add ConvertRawDataInTensorProto after loading external data to match the inline path's double-swap:

      if (onnxruntime::utils::HasExternalData(be_copy)) {
        auto be_copy_external_data{be_copy};
        ORT_RETURN_IF_ERROR(onnxruntime::utils::TensorProtoWithExternalDataToTensorProto(be_copy_external_data, {}, be_copy));
        // Swap the newly loaded LE raw_data to BE, matching what ConvertRawDataInTensorProto
        // would have done for inline data. UnpackInitializerData's ReadLittleEndian will then
        // swap it back to LE, producing the correct result.
        onnxruntime::utils::ConvertRawDataInTensorProto(be_copy);
      }

[Copilot]

Pull request overview

This PR addresses big-endian (s390x) correctness issues in ORT by standardizing raw tensor data handling (writing/reading in little-endian form where required) and adjusting affected tests/build wiring so the test suite passes without training enabled.

Changes:

• Replace direct TensorProto::set_raw_data(...) usage with onnxruntime::utils::SetRawDataInTensorProto(...) across multiple components/tests to centralize endianness handling.
• Improve test robustness on big-endian by unpacking tensor proto data via ORT utilities (e.g., UnpackTensor, ConvertRawDataInTensorProto) instead of memcpy/reinterpretation.
• Update ORT-format/flatbuffer initializer handling and unit-test build configuration to support big-endian scenarios and non-training builds.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
winml/adapter/winml_adapter_model.cpp	Use ORT helper to set TensorProto raw data with endian-awareness.
orttraining/orttraining/training_api/checkpoint.cc	Remove little-endian-only guard for training checkpoints.
orttraining/orttraining/test/graph/optimizer_graph_builder_test.cc	Read raw_data via UnpackTensor to be endian-correct.
orttraining/orttraining/core/optimizer/shape_optimizer.cc	Use endian-aware raw data setter for constant initializers.
orttraining/orttraining/core/optimizer/megatron_transformer.cc	Use endian-aware raw data setter for partitioned initializers.
orttraining/orttraining/core/optimizer/conv1d_replacement.cc	Use endian-aware raw data setter for initializer creation.
orttraining/orttraining/core/framework/checkpointing.cc	Remove big-endian “not implemented” restriction in checkpoint saving path.
onnxruntime/test/providers/nv_tensorrt_rtx/test_nv_trt_rtx_ep_util.cc	Use endian-aware raw data setter in test model building utilities.
onnxruntime/test/framework/sparse_kernels_test.cc	Convert/check raw_data in a big-endian-safe way (copy-by-value + conversion).
onnxruntime/test/framework/int2_test.cc	Use endian-aware raw data setter in Int2 round-trip test.
onnxruntime/test/framework/endian_test.cc	Use endian-aware raw data setter and add stronger assertions about conversion effects.
onnxruntime/test/flatbuffers/flatbuffer_utils_test.cc	Remove manual conversion now handled elsewhere.
onnxruntime/core/optimizer/qdq_transformer/where_dummy_dq.cc	Use endian-aware raw data setter for dummy initializer scalars.
onnxruntime/core/graph/graph_flatbuffers_utils.cc	Add big-endian handling for saving/loading ORT-format initializers and tensor dims.
onnxruntime/core/graph/graph.cc	Remove prior sparse-constant endian workaround; use endian-aware raw data setter for editor API.
onnxruntime/core/framework/tensorprotoutils.cc	Enhance raw-data setter and conversion logic; ensure sparse raw bytes are little-endian.
onnxruntime/core/framework/data_transfer_utils.h	Add byte-swapping to CopyTensorDataToByteSpan on big-endian.
cmake/onnxruntime_unittests.cmake	Ensure endian_utils is linked into tests when training is disabled.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

CopyTensorDataToByteSpan now unconditionally byte-swaps the destination buffer on big-endian hosts. This helper is used outside of checkpoint/serialization codepaths (e.g., copying device tensors into a NumPy array in the Python bindings), where callers expect a pure byte-for-byte copy in native format. Please avoid changing CopyTensorDataToByteSpan semantics globally; instead move the endian conversion to the specific serialization/checkpointing call sites that require little-endian on-disk representation, or add an explicit opt-in parameter/helper for "copy as little-endian".

[AlekseiNikiforovIBM]

Looks like a correct catch, I'll rework this part.

[Copilot]

This external-data handling in SaveInitializerOrtFormat has two issues on big-endian hosts:

1. It passes {} into TensorProtoWithExternalDataToTensorProto, but that value is forwarded as the tensor_proto_dir for ReadExternalDataForTensor, so external data may be resolved relative to the current working directory instead of the model directory.
2. After loading external bytes into be_copy.raw_data, the later UnpackInitializerData(be_copy, ...) call will invoke ReadLittleEndian and convert the bytes to native endianness before writing them into the flatbuffer, but ORT format raw_data is expected to be little-endian (LoadOrtTensorOrtFormat unpacks it with ReadLittleEndian).
   Consider reading external bytes directly into unpacked_tensor (no UnpackTensor conversion) and using the correct directory derived from model_path when resolving the external data.

Suggested change

	auto be_copy{initializer};
	onnxruntime::utils::ConvertRawDataInTensorProto(be_copy);
	// If data is external, then call to ConvertRawDataInTensorProto will not properly byteswap it.
	// A conversion using TensorProtoWithExternalDataToTensorProto does it.
	// Do it after call to ConvertRawDataInTensorProto to ensure all data is byteswapped.
	if (onnxruntime::utils::HasExternalData(be_copy)) {
	auto be_copy_external_data{be_copy};
	ORT_RETURN_IF_ERROR(onnxruntime::utils::TensorProtoWithExternalDataToTensorProto(be_copy_external_data, {}, be_copy));
	}
	ORT_RETURN_IF_ERROR(onnxruntime::utils::UnpackInitializerData(be_copy, model_path, unpacked_tensor));
	// On big-endian hosts we must be careful to preserve the little-endian layout
	// required by ORT format raw_data.
	if (onnxruntime::utils::HasExternalData(initializer)) {
	// Resolve external data relative to the model directory and read the bytes
	// directly into unpacked_tensor without any endian conversion.
	auto tensor_with_external{initializer};
	auto tensor_with_data{initializer};
	const auto model_dir = model_path.parent_path().string();
	ORT_RETURN_IF_ERROR(
	onnxruntime::utils::TensorProtoWithExternalDataToTensorProto(tensor_with_external, model_dir, tensor_with_data));
	unpacked_tensor.assign(tensor_with_data.raw_data().begin(), tensor_with_data.raw_data().end());
	} else {
	auto be_copy{initializer};
	onnxruntime::utils::ConvertRawDataInTensorProto(be_copy);
	ORT_RETURN_IF_ERROR(onnxruntime::utils::UnpackInitializerData(be_copy, model_path, unpacked_tensor));
	}

[AlekseiNikiforovIBM]

This part is reworked now

[AlekseiNikiforovIBM]

Thanks for review, I'll rework that change. It is likely that byteswapping is excessive in that place but missing in some other place.

[AlekseiNikiforovIBM]

Your finding is entirely correct if external data is in file. However, in-memory external data is actually in native endian format, i.e. big endian on big endian systems:

onnxruntime/onnxruntime/core/framework/tensorprotoutils.cc

Lines 1803 to 1815 in 65fb61b

	if (use_tensor_buffer && tensor.SizeInBytes() > kSmallTensorExternalDataThreshold) {
	// https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/graph/graph_flatbuffers_utils.cc#L302
	const auto* raw_data = tensor.DataRaw();
	ORT_ENFORCE(raw_data, "Missing raw data for tensor proto. Invalid tensor.");
	static_assert(sizeof(void*) <= sizeof(ExternalDataInfo::OFFSET_TYPE));
	// we reinterpret_cast this back to void* in tensorprotoutils.cc:GetExtDataFromTensorProto.
	// use intptr_t as OFFSET_TYPE is signed. in theory you could get a weird looking value if the address uses the
	// high bit, but that should be unlikely in a scenario where we care about memory usage enough to use this path.
	auto offset = narrow<ExternalDataInfo::OFFSET_TYPE>(reinterpret_cast<intptr_t>(raw_data));
	ExternalDataInfo::SetExternalLocationToProto(onnxruntime::utils::kTensorProtoMemoryAddressTag,
	offset, tensor.SizeInBytes(), tensor_proto);

It seems like a bad idea to byteswap this memory data in advance, so I've added byteswapping of data from file after reading it. It also revealed a couple additional byteswapping issues which I also investigated and fixed.

[tianleiwu]

GetElementSizeOfTensor Extraction (tensorprotoutils.cc)

• ⚠️ Missing STRING type returns 0, propagated to ReadLittleEndian without guard: GetElementSizeOfTensor returns 0 for unknown/unsupported types (e.g., STRING). Most callers guard with if (element_size > 1), but in ReadExternalDataForTensor for the kTensorProtoLittleEndianMemoryAddressTag path, ReadLittleEndian(element_size, src, dst) is called unconditionally. If element_size is 0, CopyLittleEndian receives 0 for element_size_in_bytes, which violates its documented precondition ("element_size_in_bytes should be greater than zero"). In practice this path is unlikely for STRING tensors (strings don't use raw external data), but a defensive check would be prudent:

  size_t element_size = GetElementSizeOfTensor(...);
  ORT_RETURN_IF(element_size == 0, "Unsupported data type for endian conversion");

GetExtDataFromTensorProto — WriteLittleEndian Misuse (tensorprotoutils.cc)

• ⚠️ WriteLittleEndian used where ReadLittleEndian is semantically correct: In the kTensorProtoLittleEndianMemoryAddressTag branch of GetExtDataFromTensorProto, the source buffer (ext_data_buf) is in little-endian format and the destination (native_data) should be in native endian format. This is a "read from LE" operation, so ReadLittleEndian should be used. The code instead calls WriteLittleEndian(element_size, src_span, dst_span), which is documented as "writes to a little-endian destination." Both functions are functionally identical (byte swap is self-inverse), so there is no runtime bug, but the naming is misleading and inconsistent with the DML provider code which correctly uses ReadLittleEndian in the same scenario.

  // Current (misleading):
  ORT_RETURN_IF_ERROR(onnxruntime::utils::WriteLittleEndian(element_size, src_span, dst_span));
  // Should be:
  ORT_RETURN_IF_ERROR(onnxruntime::utils::ReadLittleEndian(element_size, src_span, dst_span));

Save Path Byte-Swapping (graph.cc, graph_flatbuffers_utils.cc)

• ⚠️ Duplicated byte-swap boilerplate in SaveOrtTensorOrtFormat: The inline-data and external-data-writer paths in SaveOrtTensorOrtFormat both contain nearly identical byte-swap logic (allocate vector, compute element_size, WriteLittleEndian). Consider extracting into a local lambda or helper to reduce duplication. This is a readability concern, not a correctness issue.

Summary of Concerns

#	Severity	Component	Issue
1	Suggestion	GetElementSizeOfTensor	Returns 0 for unknown types; no guard before ReadLittleEndian in the kTensorProtoLittleEndianMemoryAddressTag path of ReadExternalDataForTensor.
2	Suggestion	GetExtDataFromTensorProto	WriteLittleEndian used where ReadLittleEndian is semantically correct (LE source → native dest). Functionally identical but misleading.
3	Nitpick	SaveOrtTensorOrtFormat	Duplicated byte-swap boilerplate between inline and external data writer paths could be extracted.

---

Verdict

APPROVE — The PR correctly addresses a systemic class of big-endian bugs across the entire data serialization stack. The core architectural decisions (tag split, centralized SetRawDataInTensorProto conversion, ReadExternalDataForTensor returning native endian) are sound. The ConvertRawDataInTensorProto fix for sub-word types stored in wider protobuf fields is critical and correct. The two suggestion-level concerns (missing guard for element_size=0 and WriteLittleEndian/ReadLittleEndian naming mismatch) are worth fixing but are not blockers — the former is unreachable in practice and the latter is a semantic-only issue with no runtime impact.

[tianleiwu]

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AlekseiNikiforovIBM]

I've added change to include endian headers in a couple of files due to failing pipeline indicating this issue in one of build configurations.

Edit: failing pipeline: https://github.com/microsoft/onnxruntime/actions/runs/23539560419

[tianleiwu]

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[tianleiwu]

@AlekseiNikiforovIBM, there is still build errors in DirectML builds (please also check other failed builds):

D:\a\_work\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(128,31): error C2653: 'endian': is not a class or namespace name [D:\a\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_dml.vcxproj]
Error: D:\a\_work\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(128,39): error C2065: 'native': undeclared identifier [D:\a\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_dml.vcxproj]
Error: D:\a\_work\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(128,49): error C2653: 'endian': is not a class or namespace name [D:\a\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_dml.vcxproj]
Error: D:\a\_work\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(128,57): error C2065: 'little': undeclared identifier [D:\a\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_dml.vcxproj]
Error: D:\a\_work\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp(134,63): error C2664: 'size_t onnxruntime::utils::GetElementSizeOfTensor(onnx::TensorProto_DataType)': cannot convert argument 1 from 'int32_t' to 'onnx::TensorProto_DataType' [D:\a\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_dml.vcxproj]

[tianleiwu]

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[tianleiwu]

/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).