Adds local benchmarking tools with increased flexibility

[AntoineRichard]

Description

Introduces a self-contained benchmarking framework that removes the hard dependency on isaacsim.benchmark.services.

Key additions:

• BaseIsaacLabBenchmark class with pluggable output backends (JSON, Osmo, OmniPerf)
• Measurement types (SingleMeasurement, StatisticalMeasurement, DictMeasurement, ListMeasurement, BooleanMeasurement) and metadata types for structured recording
• BenchmarkMonitor context manager for async resource monitoring during blocking RL training loops.
• System recorders for CPU, GPU, memory, and version info collection
• MethodBenchmarkRunner for method-level micro-benchmarks
• Shell scripts for CI/CD integration (run_non_rl_benchmarks.sh, run_training_benchmarks.sh, run_physx_benchmarks.sh)

Updated:

• PhysX micro-benchmarks to use the new framework
• Documentation and changelogs

Type of change

• New feature (non-breaking change which adds functionality)
• Documentation update

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Overview

Greptile Summary

This PR introduces a self-contained benchmarking framework that removes the hard dependency on isaacsim.benchmark.services, significantly improving flexibility and modularity.

Key additions:

• BaseIsaacLabBenchmark class with pluggable output backends (JSON, Osmo, OmniPerf)
• Comprehensive measurement types: SingleMeasurement, StatisticalMeasurement, BooleanMeasurement, DictMeasurement, ListMeasurement
• BenchmarkMonitor context manager for async resource monitoring during blocking RL training loops
• System recorders: CPUInfoRecorder, GPUInfoRecorder, MemoryInfoRecorder, VersionInfoRecorder with Welford's algorithm for online statistics
• MethodBenchmarkRunner for method-level micro-benchmarks with warmup/timing support
• Shell scripts for CI/CD integration

Changes:

• PhysX micro-benchmarks refactored to use new framework
• RL training benchmarks updated to use BenchmarkMonitor for continuous resource monitoring
• Removed old benchmark_io.py file (560 lines deleted)

Issues found:

• Shell script run_non_rl_benchmarks.sh has a variable name bug ($task should be $TASK) that will cause the camera condition check to never match

Confidence Score: 4/5

• Safe to merge after fixing the shell script variable name bug
• Well-structured framework with comprehensive tests and clear architecture. The only critical issue is a shell script bug that will prevent camera-enabled benchmarks from running correctly. Code quality is high with proper error handling, statistical algorithms (Welford), and good separation of concerns.
• scripts/benchmarks/run_non_rl_benchmarks.sh - Variable name case mismatch will break camera detection logic

Important Files Changed

Filename	Overview
scripts/benchmarks/run_non_rl_benchmarks.sh	New shell script with variable name case mismatch bug
source/isaaclab/isaaclab/test/benchmark/benchmark_core.py	Core benchmark class refactored with pluggable backends and improved modularity
source/isaaclab/isaaclab/test/benchmark/backends.py	New pluggable backend system with JSON, Osmo, and OmniPerf support
source/isaaclab/isaaclab/test/benchmark/measurements.py	Comprehensive measurement types (Single, Statistical, Boolean, Dict, List) and metadata classes
source/isaaclab/isaaclab/test/benchmark/benchmark_monitor.py	Context manager for async resource monitoring during blocking RL training loops
source/isaaclab/isaaclab/test/benchmark/method_benchmark.py	Method-level micro-benchmark runner with warmup, timing, and statistical measurements

Sequence Diagram

sequenceDiagram
    participant User
    participant Benchmark as BaseIsaacLabBenchmark
    participant Backend as MetricsBackend
    participant Recorders as System Recorders
    participant Monitor as BenchmarkMonitor
    participant Training as RL Training Loop

    User->>Benchmark: Initialize(backend_type, output_path)
    Benchmark->>Backend: get_instance(backend_type)
    Backend-->>Benchmark: Return backend (JSON/Osmo/OmniPerf)
    Benchmark->>Recorders: Initialize CPUInfo, GPUInfo, MemoryInfo, VersionInfo
    Recorders-->>Benchmark: Recorders ready
    
    alt Blocking RL Training
        User->>Monitor: with BenchmarkMonitor(benchmark)
        Monitor->>Monitor: Start background thread
        loop Training iterations
            Training->>Training: runner.learn() [blocking]
            Monitor->>Recorders: update_manual_recorders() [async, every 1s]
            Recorders->>Recorders: Collect CPU/GPU/Memory stats using Welford algorithm
        end
        Training-->>Monitor: Training complete
        Monitor->>Monitor: Stop background thread
    else Non-blocking workflow
        User->>Benchmark: update_manual_recorders() [manual calls]
        Benchmark->>Recorders: update()
    end
    
    User->>Benchmark: add_measurement(phase_name, measurement)
    Benchmark->>Benchmark: Store in TestPhase
    
    User->>Benchmark: finalize()
    Benchmark->>Recorders: get_data()
    Recorders-->>Benchmark: Return measurements & metadata
    Benchmark->>Backend: add_metrics(test_phase)
    Backend->>Backend: Accumulate phases
    Benchmark->>Backend: finalize(output_path, filename)
    Backend->>Backend: Write JSON/Osmo/OmniPerf file
    Backend-->>User: Results written to file

Loading

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[kellyguo11]

Looks like some mismatch in names between the recorders and test cases, seeing failures in a couple of tests:

| /workspace/isaaclab/source/isaaclab_physx/test/assets/test_articulation.py | FAILED | 0.00 | -1/0 |
| /workspace/isaaclab/source/isaaclab/test/benchmark/test_benchmark_core.py | FAILED | 0.62 | 14/19 |
| /workspace/isaaclab/source/isaaclab/test/benchmark/test_recorders.py | FAILED | 2.95 | 26/43 |