Plugin EP event profiling APIs

[adrianlizarraga]

Description

TLDR

This PR ports the existing C++ EpProfiler interfaces used by provider-bridge EPs to the binary-stable C APIs for plugin EPs. It introduces C/C++ APIs for creating/querying profiling events, a container for appending EP events, and callback hooks (StartEvent/StopEvent) that give EPs access to ORT event metadata in real-time.

Changes to the original C++ API

The original EpProfiler C++ interface was adapted for the C API with the following intentional changes:

1. StartProfiling now receives an offset indicating the elapsed time since profiling started, as opposed to receiving an absolute/epoch-dependent profiling start time. This prevents EPs from having to do epoch conversions. Credit to @edgchen1 for the idea.
2. StartEvent/StopEvent receive an absolute, epoch-based correlation ID (ort_event_correlation_id) instead of a relative ORT event ID. The PluginEpProfiler bridge layer automatically converts the C++ relative_ort_event_id (microseconds since profiling start) to an absolute ort_event_correlation_id by adding the epoch-based profiling start time. This means plugin EPs can use the correlation ID directly with profiling utilities like CUPTI or ROCTracer without computing the conversion themselves.
3. StopEvent now receives the completed ORT event as a parameter. This allows EPs to optionally inspect ORT event metadata (e.g., op_name, event_name) at the time the event ends, facilitating annotation of correlated EP events.
4. EndProfiling only allows EPs to append events (via OrtProfilingEventsContainer), not read or modify the full events array. This is motivated by:
   • Prevent any one EP from modifying events generated by ORT or another EP.
   • Certain EPs (VitisAI and WebGPU) already only append events without reading the entire events array.
   • The CUDA EP reads the entire events array solely to merge/sort its own EP events next to correlated ORT events and add parent_name/op_name metadata. However:
     • Merging/sorting is mostly unnecessary since trace viewers that load these files do their own event sorting.
     • This merging/sorting step was previously required to augment CUDA EP events with metadata from the correlated ORT event. However, that can now be obtained more simply via the new StopEvent parameter that provides the EP with the full correlated ORT event.
     • The merge algorithm used by CUDA EP incorrectly assumes ORT events are sorted by non-decreasing start time, but they are actually sorted by non-decreasing end time (also see Session-aware and thread-safe CUDA profiler #13706 (comment)). Fixing this would require sorting the entire Events array before asking a provider-bridge EP to merge in its events into the global events array. Not sure this is worth the runtime cost.

Naming conventions for ORT event IDs

• C++ EpProfiler interface (existing): Uses relative_ort_event_id — a timestamp offset in microseconds relative to profiling start.
• C API OrtEpProfilerImpl (new in this PR): Uses ort_event_correlation_id — an absolute, epoch-based timestamp in microseconds computed from std::chrono::high_resolution_clock (platform-defined epoch). Unique across concurrent profiling sessions within the same process.
• Conversion: The PluginEpProfiler bridge class (in ep_event_profiling.cc) performs ort_event_correlation_id = relative_ort_event_id + profiling_start_time_epoch_us_, mirroring the pattern in GPUTracerManager::PushCorrelation.

New C APIs

API	Description
CreateProfilingEvent	Create a profiling event with category, process/thread IDs, name, timestamp, duration, and key-value args
ReleaseProfilingEvent	Release a profiling event
ProfilingEvent_GetCategory	Get event category (SESSION, NODE, KERNEL, API)
ProfilingEvent_GetName	Get event name
ProfilingEvent_GetTimestampUs	Get event start timestamp (µs)
ProfilingEvent_GetDurationUs	Get event duration (µs)
ProfilingEvent_GetArgValue	Get an event argument value by key
ProfilingEventsContainer_AddEvents	Append an array of EP events to the output container
OrtEp::CreateProfiler	Returns an instance of the EP's profiler implementation
OrtEpProfilerImpl::StartProfiling	Called by ORT to start a profiling session. Receives elapsed time offset (ns) since ORT profiling started
OrtEpProfilerImpl::StartEvent	Called by ORT to notify that an ORT event has started. Receives an absolute ort_event_correlation_id
OrtEpProfilerImpl::StopEvent	Called by ORT to notify that an ORT event has ended. Receives the same ort_event_correlation_id and ORT event metadata
OrtEpProfilerImpl::EndProfiling	Called by ORT to end the profiling session and collect EP events into the output container
OrtEpProfilerImpl::Release	Release the profiler instance

New C++ wrapper classes

Class	Description
Ort::ConstProfilingEvent	Non-owning const wrapper for reading fields from an OrtProfilingEvent (e.g., in StopEvent)
Ort::ProfilingEvent	Owning wrapper that creates and manages an OrtProfilingEvent (e.g., for EndProfiling)
Ort::UnownedProfilingEventsContainer	Non-owning wrapper for adding events to an OrtProfilingEventsContainer during EndProfiling

Example EP profiling implementation

This PR updates an example plugin EP to use the new profiling APIs:

• Plugin EP code: test/autoep/library/example_plugin_ep_kernel_registry
• OrtEpProfilerImpl implementation: ep_profiling.h / ep_profiling.cc
• OrtEp::CreateProfiler() implementation: ep.cc

Existing bugs found

Not fixed in this PR.

• The merge algorithm used by CUDA EP incorrectly assumes ORT events are sorted by non-decreasing start time, but they are actually sorted by non-decreasing end time (also see Session-aware and thread-safe CUDA profiler #13706 (comment)).
• Run profilers do not handle subgraphs (e.g., subgraph of a control-flow operator). Has been the case since run profilers were introduced.

Motivation and Context

Allows plugin EPs to generate profiling events, further closing the functionality gap between provider-bridge EPs and plugin EPs.

[github-actions]

You can commit the suggested changes from lintrunner.

[Copilot]

Pull request overview

Adds draft profiling support for plugin Execution Providers (EPs) by extending the EP C API with a profiler interface and an event-ingestion mechanism, and wiring this into the plugin EP host/provider implementation plus an example/test EP.

Changes:

• Introduces OrtEpProfilerImpl / OrtEpProfilingEvent APIs and extends OrtEp/OrtEpApi for plugin EP profiling.
• Implements host-side wrapping (profiling::EpProfiler) for plugin-provided profilers and adds an ORT-side events container (OrtEpProfilingEventsContainer) with an AddEvents API.
• Updates the example kernel-registry plugin EP to emit profiling events and adds an autoep test validating the profile output.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
onnxruntime/test/autoep/test_execution.cc	Adds a new test that enables profiling and checks plugin EP events appear in the profile output.
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep.h	Adds GetProfiler hook to the example plugin EP.
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep.cc	Implements an example OrtEpProfilerImpl that reports events via the new container API.
onnxruntime/core/session/plugin_ep/ep_profiling_events_container.h	Adds the concrete definition for the previously-opaque profiling events container.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.h	Adds GetProfiler() override for plugin EP provider.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc	Wraps plugin OrtEpProfilerImpl into profiling::EpProfiler and integrates with ORT profiling.
onnxruntime/core/session/plugin_ep/ep_api.h	Adds EpProfilingEventsContainer_AddEvents to the plugin EP API surface.
onnxruntime/core/session/plugin_ep/ep_api.cc	Implements EpProfilingEventsContainer_AddEvents and appends it to the versioned OrtEpApi table.
include/onnxruntime/core/session/onnxruntime_ep_c_api.h	Defines public C API types for profiling events and the plugin EP profiler interface; adds OrtEp::GetProfiler and OrtEpApi::EpProfilingEventsContainer_AddEvents.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[github-actions]

You can commit the suggested changes from lintrunner.

[Copilot]

Pull request overview

This PR drafts profiling support for plugin Execution Providers (EPs) by adding a public EP profiling interface (C API + C++ wrapper) and wiring it into the plugin EP execution provider path, along with an example plugin implementation and a validation test.

Changes:

• Introduces public plugin EP profiling types/callbacks (OrtEpProfilerImpl) and EP-side event reporting helpers (CreateEpProfilingEvent, EpProfilingEventsContainer_AddEvents).
• Adds ORT-side wrapper (PluginEpProfiler) to invoke plugin profiler callbacks and merge EP events into ORT’s profiling timeline.
• Implements and tests profiling in the example plugin EP kernel registry, validating that EP kernel events show up in the profiling output.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
onnxruntime/test/autoep/test_execution.cc	Adds a unit test that enables profiling and checks the output contains plugin EP events.
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/kernels/binary_op.cc	Records per-kernel timing events and reports them through the example EP profiling manager.
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep_profiling.h	Declares example EP profiler implementation and the singleton event manager used by kernels.
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep_profiling.cc	Implements event tracking, correlation stack handling, and event conversion into ORT profiling events.
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep.h	Extends the example EP to expose profiler state (active profiler client id).
onnxruntime/test/autoep/library/example_plugin_ep_kernel_registry/ep.cc	Implements OrtEp::GetProfiler for the example EP.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.h	Extends PluginExecutionProvider to provide an EP profiler instance.
onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc	Wires GetProfiler() for plugin EPs and refactors logger fallback handling.
onnxruntime/core/session/plugin_ep/ep_event_profiling.h	Defines ORT-side opaque EP profiling event/container and PluginEpProfiler wrapper interface.
onnxruntime/core/session/plugin_ep/ep_event_profiling.cc	Implements event merging and the PluginEpProfiler wrapper calling plugin callbacks.
onnxruntime/core/session/plugin_ep/ep_api.h	Exposes new EP API entrypoints for creating/releasing EP profiling events and adding them to a container.
onnxruntime/core/session/plugin_ep/ep_api.cc	Implements the new EP profiling event/container API functions and appends them to the EP API table.
onnxruntime/core/framework/error_code_helper.h	Adds ORT_API_RETURN_IF helper macro for concise C API argument validation.
onnxruntime/core/common/profiler.cc	Sorts ORT events by start timestamp before merging EP events.
include/onnxruntime/core/session/onnxruntime_ep_c_api.h	Adds public EP profiling C API declarations and documentation.
include/onnxruntime/core/common/profiler_common.h	Expands documentation for the C++ profiling::EpProfiler interface.
cmake/onnxruntime_unittests.cmake	Adds the example EP profiling sources to unit test build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tianleiwu]

Please resolve merge conflicts.

Summary

This PR ports the C++ EpProfiler interface to a binary-stable C API for plugin EPs, with well-considered design improvements (epoch-independent offsets, absolute correlation IDs, append-only event containers). The architecture is clean with proper layering: C API → C++ wrappers → bridge layer. The code is overall high quality with thorough validation, ABI slot assertions, and comprehensive tests covering both session and run profiling modes. A few concerns around enum synchronization guarantees and a minor UB-by-the-letter in the C++ wrapper reinterpret_cast are noted below.

---

Review

1. C API Surface (onnxruntime_ep_c_api.h)

Positive:

• The OrtEpProfilerImpl struct documentation is exceptionally thorough—it distinguishes session vs. run profiling concurrency semantics, threading guarantees for StartEvent/StopEvent, and the meaning of the absolute correlation ID. This level of documentation is rare and invaluable for EP authors.
• The OrtProfilingEventsContainer being append-only is a sound design choice that prevents one EP from corrupting another EP's events or ORT's own events.
• StartEvent/StopEvent being optional (NULL-able) is good for EPs that don't need correlation tracking.

Concern:

• ⚠️ Enum synchronization is enforced only by comment: The C++ EventCategory enum in profiler_common.h carries a "keep in sync with OrtProfilingEventCategory" comment, but there is no compile-time enforcement. A static_assert verifying value equality would prevent silent drift:

  static_assert(static_cast<int>(OrtProfilingEventCategory_SESSION) == static_cast<int>(profiling::EventCategory::SESSION_EVENT));
  static_assert(static_cast<int>(OrtProfilingEventCategory_NODE) == static_cast<int>(profiling::EventCategory::NODE_EVENT));
  static_assert(static_cast<int>(OrtProfilingEventCategory_KERNEL) == static_cast<int>(profiling::EventCategory::KERNEL_EVENT));
  static_assert(static_cast<int>(OrtProfilingEventCategory_API) == static_cast<int>(profiling::EventCategory::API_EVENT));

2. C++ Wrappers (onnxruntime_cxx_api.h, onnxruntime_cxx_inline.h)

Positive:

• The ConstProfilingEvent / ProfilingEvent / UnownedProfilingEventsContainer trio follows the established ORT wrapper patterns (Const*, owning, unowned). Consistent with the rest of the API.
• The ProfilingEvent constructor overload accepting std::unordered_map<std::string, std::string> provides a natural C++ interface while the raw const char** overload supports zero-copy scenarios.

Concern:

• ⚠️ reinterpret_cast for vector of wrappers is formally UB: The AddEvents(const std::vector<ProfilingEvent>&) overload casts the vector data pointer to OrtProfilingEvent**:

  static_assert(sizeof(ProfilingEvent) == sizeof(OrtProfilingEvent*) && ...);
  const auto* event_ptrs = reinterpret_cast<const OrtProfilingEvent* const*>(events.data());

  While the static_assert guards the layout assumption and this pattern is used elsewhere in ORT, it is technically a strict-aliasing violation under C++ rules. A safe alternative would be to build a temporary pointer array:

  std::vector<const OrtProfilingEvent*> ptrs(events.size());
  for (size_t i = 0; i < events.size(); ++i) ptrs[i] = events[i];
  return AddEvents(ptrs.data(), ptrs.size());

  This is a Suggestion rather than a blocker, as the existing pattern is well-established in ORT.

3. C API Implementation (ep_api.cc)

Positive:

• Thorough input validation in CreateProfilingEvent: validates output pointer, event name, category range, arg array consistency (non-null keys/values when num_args > 0), and per-element null checks. This is exemplary defensive coding for a C API boundary.
• ProfilingEventsContainer_AddEvents validates all events for null BEFORE modifying the container, ensuring atomicity on error.
• ABI slot assertions are properly maintained (static_assert(offsetof(...) == 57, ...)).

Concern:

• ⚠️ ORT_API_RETURN_IF macro placement: The new ORT_API_RETURN_IF macro in error_code_helper.h is a useful utility, but it includes core/common/make_string.h at header level. This adds make_string.h as a transitive dependency for all consumers of error_code_helper.h. If make_string.h is already widely included this is fine; otherwise consider a forward-declared helper or noting the include impact. (Nitpick)

4. Bridge Layer (ep_event_profiling.cc, ep_event_profiling.h)

Positive:

• The PluginEpProfiler::Create() factory with PrivateTag is a clean pattern that ensures validation always runs before construction—prevents partially constructed objects.
• The epoch-independent timestamp computation (offset_ns = now() - profiling_start_time) is an elegant solution that eliminates clock epoch mismatches between ORT and the EP.
• Proper error handling throughout: all profiling callback errors are logged but do not throw, preserving inference execution.
• The ToOpaqueProfilingEvent / FromOpaqueProfilingEvent helpers are clean type-puns that centralize the cast in one place.

Concern:

• ⚠️ Version-check leak path: In PluginEpProfiler::Create(), if ort_version_supported < 25, the profiler returned by the EP is not released because Release itself may not exist at that version. A comment documents this ambiguity, which is good. However, the caller PluginExecutionProvider::GetProfiler() already gates on ort_ep_->ort_version_supported < 25 before calling CreateProfiler, making this dead code in practice. Consider either removing the redundant check or adding a brief comment to Create() noting the caller's pre-check makes this unreachable. (Nitpick)

5. Profiler Base Class Changes (profiler_common.h, profiler.cc)

Positive:

• Enriching Stop() with const EventRecord& is well-motivated—it lets EP profilers annotate their events with ORT event metadata (op_name, event_name) at stop time, eliminating the need for the post-hoc merge algorithm used by CUDA EP.
• The detailed doc comments on Start()/Stop() explaining relative vs. absolute IDs and the conversion formula are excellent. These will save plugin EP authors significant debugging time.
• Moving the Stop() call above event storage in profiler.cc is correct because the event hasn't been moved yet at that point. The const EventRecord& parameter ensures the data is available during the callback.

Concern:

• None for this component.

6. Existing EP Profiler Updates (gpu_profiler_common.h, cuda_profiler.h, vitisai_profiler.h, webgpu_profiler.h)

Positive:

• The signature changes are mechanical and correct: void Stop(uint64_t) → void Stop(uint64_t, const EventRecord&) with the EventRecord parameter unused in these existing implementations to preserve current behavior.
• The new TODO comment in gpu_profiler_common.h documenting the incorrect sort-order assumption in the merge algorithm is a helpful breadcrumb for future work.

Concern:

• None. Clean signature adaptation.

7. Example EP Profiler (ep_profiling.cc, ep_profiling.h)

Positive:

• The EpEventManager singleton with per-profiler state and thread-local correlation tracking is a clear demonstration of the intended usage pattern. It correctly handles:
  • Multiple concurrent profiling sessions (each with its own profiler_id)
  • Inter-op parallelism (thread-local ORT event stacks)
  • Event annotation by filtering on thread_id during PopOrtEvent
• Event timestamp computation in EndProfilingImpl correctly combines the ORT-provided offset with the EP's local elapsed time—a faithful implementation of the documented formula.

Concern:

• ⚠️ Mutex contention in hot path: EpEventManager holds a global mutex for AddEpEvent, PushOrtEvent, and PopOrtEvent. In Compute() (binary_op.cc), AddEpEvent is called under this mutex on every kernel invocation when profiling is active. For a real EP with high throughput, this serialized access to the global events vector could become a bottleneck. For test/example code this is completely fine, but the example might benefit from a brief comment noting that a production EP should use a more scalable approach (e.g., per-thread event collection). (Suggestion)

8. Kernel Integration (binary_op.cc)

Positive:

• Conditional profiling (only activating when GetActiveProfilerId() returns a value) ensures zero overhead on the hot path when profiling is disabled.
• Using std::chrono::high_resolution_clock consistently with the profiler start point eliminates clock-source mismatches.

Concern:

• None.

9. Tests (test_execution.cc, ep_plugin_provider_test.cc, inference_session_test.cc)

Positive:

• The end-to-end profiling tests (KernelPluginEp_SessionProfiling, KernelPluginEp_RunProfiling) validate the full pipeline: run model → parse profile JSON → find EP event → verify category/timestamps/duration/args → verify parent ORT event containment. This is thorough coverage.
• The unit tests for C API (CreateProfilingEvent_AllCategories, *_NullOutput, *_NullKey) and C++ wrappers validate all error paths and both constructor overloads.
• The inference_session_test.cc fix to CheckRunProfilerWithStartProfile removes the fragile line-count-dependent assertions and replaces them with a proper search. Good defensive test improvement.
• The run-profiling test honestly acknowledges the subgraph limitation as a TODO with commented-out assertions rather than asserting wrong behavior.

Concern:

• ⚠️ Run profiling file discovery is fragile: The run profiling test finds the profile file by scanning the current directory for the newest file matching the prefix. In a parallel test execution environment, two separate test processes could produce files with near-identical timestamps, causing the wrong file to be selected. The cleanup gsl::finally mitigates stale file risk, but the window exists. Consider using a unique subdirectory per test invocation to isolate profile output. (Suggestion—may be an ORT test infra limitation)
• ⚠️ profile_file_prefix type change: The change from "ort_run_profile_test" to ORT_TSTR("ort_run_profile_test") in inference_session_test.cc is a type fix. Verify that RunOptions::profile_file_prefix was recently changed from std::string to std::basic_string<ORTCHAR_T> or similar—otherwise this would be a compile error. This appears to be correct but is worth confirming. (Nitpick)

10. Build System (onnxruntime_unittests.cmake)

Positive:

• New source files (ep_profiling.h, ep_profiling.cc) are correctly added to the example plugin EP's build target, keeping them adjacent to the other EP source files.

Concern:

• None.

---

Summary of Concerns

#	Severity	Component	Issue
1	Suggestion	C API (onnxruntime_ep_c_api.h)	Add static_assert to enforce enum value equality between C++ EventCategory and C OrtProfilingEventCategory
2	Suggestion	C++ wrappers (onnxruntime_cxx_inline.h)	reinterpret_cast in AddEvents is technically strict-aliasing UB; consider a temporary pointer array
3	Nitpick	C API impl (error_code_helper.h)	New #include "make_string.h" adds a transitive dependency via ORT_API_RETURN_IF
4	Nitpick	Bridge layer (ep_event_profiling.cc)	Version-check code path in Create() is unreachable given caller's pre-check; add comment or remove
5	Suggestion	Example EP (ep_profiling.cc)	Global mutex in EpEventManager could bottleneck real EPs; add comment noting this is example-only
6	Suggestion	Tests (test_execution.cc)	Run profiling file discovery by timestamp is fragile under parallel test execution
7	Nitpick	Tests (inference_session_test.cc)	Confirm profile_file_prefix type change to ORT_TSTR() matches upstream RunOptions field type

---

Verdict

APPROVE — This is a well-designed, thoroughly documented, and carefully validated API extension. The concerns are suggestions and nitpicks; none represent correctness bugs or security issues. The code correctly maintains ABI stability, validates all inputs at the C API boundary, handles error paths gracefully, and provides comprehensive test coverage.

[tianleiwu]

Approved. The overall API shape looks solid.