HMD Optical Hand tracking and MANUS automatic installation

docs/source/device/index.rst

@@ -19,3 +19,4 @@ See the `Plugins directory <https://github.com/NVIDIA/IsaacTeleop/tree/main/src/
 
    trackers
    add_device
+   manus

docs/source/device/manus.rst

@@ -0,0 +1,183 @@
+..
+   SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+   SPDX-License-Identifier: Apache-2.0
+
+Manus Gloves
+============
+
+A Linux-only plugin for integrating `Manus <https://www.manus-meta.com/>`_ gloves
+into the Isaac Teleop framework. It provides full hand-joint tracking via the
+Manus SDK and injects the resulting poses into the OpenXR hand-tracking layer so
+any downstream retargeter can consume them transparently.
+
+Components
+----------
+
+- **Core library** (``manus_plugin_core``) — wraps the Manus SDK
+  (``libIsaacTeleopPluginsManus.so``) and exposes per-joint tracking data.
+- **Plugin executable** (``manus_hand_plugin``) — the main plugin binary that
+  integrates with the Teleop system via CloudXR / OpenXR.
+- **CLI tool** (``manus_hand_tracker_printer``) — a standalone diagnostic tool
+  that prints tracked joint data to the terminal and opens a real-time
+  **MANUS Data Visualizer** window showing the hand skeleton from two orthographic
+  views per hand.
+
+Prerequisites
+-------------
+
+- **Linux** — x86_64 (tested on Ubuntu 22.04 / 24.04).
+- **Manus SDK** for Linux — downloaded automatically by the install script.
+- **System dependencies** — the install script installs required packages automatically.
+
+Installation
+------------
+
+Automated (recommended)
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The install script handles SDK download, dependency installation, and building:
+
+.. code-block:: bash
+
+   cd src/plugins/manus
+   ./install_manus.sh
+
+The script will:
+
+1. Install the required system packages for MANUS Core Integrated.
+2. Download MANUS SDK v3.1.1.
+3. Extract and place the SDK in the correct location.
+4. Build the plugin and the diagnostic tool
+
+Manual
+~~~~~~
+
+If you prefer to install manually:
+
+1. Download the MANUS Core SDK from
+   `MANUS Downloads <https://docs.manus-meta.com/3.1.1/Resources/>`_.
+2. Extract and place the ``ManusSDK`` folder inside ``src/plugins/manus/``, or
+   point CMake at a different path by setting ``MANUS_SDK_ROOT``.
+3. Follow the
+   `MANUS Getting Started guide for Linux <https://docs.manus-meta.com/3.1.1/Plugins/SDK/Linux/>`_
+   to install the dependencies and configure device permissions.
+
+Expected directory layout after placing the SDK:
+
+.. code-block:: text
+
+   src/plugins/manus/
+     app/
+       main.cpp
+     core/
+       manus_hand_tracking_plugin.cpp
+     inc/
+       core/
+         manus_hand_tracking_plugin.hpp
+     tools/
+       manus_hand_tracker_printer.cpp
+     ManusSDK/        <-- placed here
+       include/
+       lib/
+
+Then build from the root:
+
+.. code-block:: bash
+
+   cd ../../..  # navigate to root
+   cmake -S . -B build
+   cmake --build build --target manus_hand_plugin manus_hand_tracker_printer -j
+   cmake --install build --component manus
+
+Running the Plugin
+------------------
+
+1. Set up the CloudXR environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Source the CloudXR environment and start the runtime before running the plugin:
+
+.. code-block:: bash
+
+   export NV_CXR_RUNTIME_DIR=~/.cloudxr/run
+   export XR_RUNTIME_JSON=~/.cloudxr/openxr_cloudxr.json
+
+2. Verify with the CLI tool
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Verify that the gloves are working using the CLI tool:
+
+.. code-block:: bash
+
+   ./build/bin/manus_hand_tracker_printer
+
+The tool prints joint positions to the terminal and opens a **MANUS Data
+Visualizer** window showing a top-down and side view of each hand.
+
+3. Run the plugin
+~~~~~~~~~~~~~~~~~~
+
+The plugin is installed to the `install` directory, please ensure the CLI tool is not running when running the plugin.
+
+.. code-block:: bash
+
+   ./install/plugins/manus/manus_hand_plugin
+
+Wrist Positioning — Controllers vs Optical Hand Tracking
+---------------------------------------------------------
+
+Two sources are available for positioning the Manus gloves in 3D space:
+
+- **Controller adapters** — attach Quest 3 controllers to the Manus Universal
+  Mount on the back of the glove. The controller pose drives wrist placement.
+- **Optical hand tracking** — use the HMD's built-in optical hand tracking to
+  position the hands. No physical controller adapter required.
+
+The plugin selects the source automatically at runtime: optical hand tracking is
+preferred when ``XR_MNDX_xdev_space`` is supported and the runtime reports an
+actively tracked wrist pose; otherwise it falls back to the controller pose.
+
+.. note::
+
+   When using controller adapters it is advisable to disable the HMD's automatic
+   hand-tracking–to–controller switching to avoid unexpected source changes
+   mid-session.
+
+Troubleshooting
+---------------
+
+.. list-table::
+   :widths: 40 60
+   :header-rows: 1
+
+   * - Symptom
+     - Resolution
+   * - SDK download fails
+     - Check your internet connection and re-run the install script.
+   * - Manus SDK not found at build time
+     - With manual installation, ensure ``ManusSDK`` is inside
+       ``src/plugins/manus/`` or set ``MANUS_SDK_ROOT`` to your installation path.
+   * - Manus SDK not found at runtime
+     - The build configures RPATH automatically. If you moved the SDK after
+       building, set ``LD_LIBRARY_PATH`` to its ``lib/`` directory.
+   * - No data received
+     - Ensure Manus Core is running and the gloves are connected and calibrated.
+   * - CloudXR runtime errors
+     - Make sure ``scripts/setup_cloudxr_env.sh`` has been sourced before running.
+   * - Permission denied for USB devices
+     - The install script configures udev rules. If the rules were not reloaded,
+       run:
+
+       .. code-block:: bash
+
+          sudo udevadm control --reload-rules
+          sudo udevadm trigger
+
+       Then reconnect your Manus devices.
+
+License
+-------
+
+Source files are covered by their stated licenses (Apache-2.0). The Manus SDK is
+proprietary to Manus and is subject to its own license; it is **not** redistributed
+by this project.

examples/native_openxr/xdev_list/main.cpp

@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 #include "common/main_helpers.hpp"
@@ -159,16 +159,16 @@ class XDevListApp : public HeadlessApp
             std::string serial_str = properties.serial ? properties.serial : "";
             if (serial_str == "Head Device (0)" || serial_str == "Head Device (1)")
             {
-                std::cout << "[CREATE HAND] XDev ID=" << xdevId << " Name=\"" << properties.name << "\""
-                          << " Serial=\"" << serial_str << "\"" << std::endl;
+                std::cout << "[CREATE HAND] XDev ID=" << xdevId << " Name=\"" << properties.name << "\"" << " Serial=\""
+                          << serial_str << "\"" << std::endl;
 
                 XrHandEXT hand = (serial_str == "Head Device (1)") ? XR_HAND_RIGHT_EXT : XR_HAND_LEFT_EXT;
                 create_hand_tracker_from_xdev(openxr_bundle, xdevList, xdevId, hand);
             }
             else
             {
-                std::cout << "[SKIP] XDev ID=" << xdevId << " Name=\"" << properties.name << "\""
-                          << " Serial=\"" << (properties.serial ? properties.serial : "") << "\"" << std::endl;
+                std::cout << "[SKIP] XDev ID=" << xdevId << " Name=\"" << properties.name << "\"" << " Serial=\""
+                          << (properties.serial ? properties.serial : "") << "\"" << std::endl;
             }
         }
 

examples/oxr/cpp/oxr_session_sharing.cpp

@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 #include <deviceio_session/deviceio_session.hpp>
@@ -85,8 +85,7 @@ try
 
         if (i % 3 == 0)
         {
-            std::cout << "Frame " << i << ": "
-                      << "Hands=" << (left_tracked.data ? "ACTIVE" : "INACTIVE") << " | "
+            std::cout << "Frame " << i << ": " << "Hands=" << (left_tracked.data ? "ACTIVE" : "INACTIVE") << " | "
                       << "Head=" << ((head_tracked.data && head_tracked.data->is_valid) ? "VALID" : "INVALID");
             if (head_tracked.data && head_tracked.data->is_valid && head_tracked.data->pose)
             {

src/core/deviceio_trackers/python/tracker_bindings.cpp

@@ -29,34 +29,31 @@ PYBIND11_MODULE(_deviceio_trackers, m)
         .def(
             "get_left_hand",
             [](const core::HandTracker& self, const core::ITrackerSession& session) -> core::HandPoseTrackedT
-            { return self.get_left_hand(session); },
-            py::arg("session"))
+            { return self.get_left_hand(session); }, py::arg("session"))
         .def(
             "get_right_hand",
             [](const core::HandTracker& self, const core::ITrackerSession& session) -> core::HandPoseTrackedT
-            { return self.get_right_hand(session); },
-            py::arg("session"));
+            { return self.get_right_hand(session); }, py::arg("session"))
+        .def_static("get_joint_name", &core::HandTracker::get_joint_name);
 
     py::class_<core::HeadTracker, core::ITracker, std::shared_ptr<core::HeadTracker>>(m, "HeadTracker")
         .def(py::init<>())
         .def(
-            "get_head",
-            [](const core::HeadTracker& self, const core::ITrackerSession& session) -> core::HeadPoseTrackedT
-            { return self.get_head(session); },
-            py::arg("session"));
+            "get_head", [](const core::HeadTracker& self, const core::ITrackerSession& session) -> core::HeadPoseTrackedT
+            { return self.get_head(session); }, py::arg("session"));
 
     py::class_<core::ControllerTracker, core::ITracker, std::shared_ptr<core::ControllerTracker>>(m, "ControllerTracker")
         .def(py::init<>())
         .def(
             "get_left_controller",
             [](const core::ControllerTracker& self, const core::ITrackerSession& session) -> core::ControllerSnapshotTrackedT
-            { return self.get_left_controller(session); },
-            py::arg("session"), "Get the left controller tracked state (data is None if inactive)")
+            { return self.get_left_controller(session); }, py::arg("session"),
+            "Get the left controller tracked state (data is None if inactive)")
         .def(
             "get_right_controller",
             [](const core::ControllerTracker& self, const core::ITrackerSession& session) -> core::ControllerSnapshotTrackedT
-            { return self.get_right_controller(session); },
-            py::arg("session"), "Get the right controller tracked state (data is None if inactive)");
+            { return self.get_right_controller(session); }, py::arg("session"),
+            "Get the right controller tracked state (data is None if inactive)");
 
     py::class_<core::FrameMetadataTrackerOak, core::ITracker, std::shared_ptr<core::FrameMetadataTrackerOak>>(
         m, "FrameMetadataTrackerOak")
@@ -92,8 +89,8 @@ PYBIND11_MODULE(_deviceio_trackers, m)
         .def(
             "get_body_pose",
             [](const core::FullBodyTrackerPico& self, const core::ITrackerSession& session) -> core::FullBodyPosePicoTrackedT
-            { return self.get_body_pose(session); },
-            py::arg("session"), "Get full body pose tracked state (data is None if inactive)");
+            { return self.get_body_pose(session); }, py::arg("session"),
+            "Get full body pose tracked state (data is None if inactive)");
 
     m.attr("NUM_JOINTS") = static_cast<int>(core::HandJoint_NUM_JOINTS);
     m.attr("JOINT_PALM") = static_cast<int>(core::HandJoint_PALM);

src/core/live_trackers/cpp/live_controller_tracker_impl.cpp

@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 #include "live_controller_tracker_impl.hpp"
@@ -254,9 +254,8 @@ LiveControllerTrackerImpl::LiveControllerTrackerImpl(const OpenXRSessionHandles&
 {
     // Suggest interaction profile bindings (chained to this action context)
     std::vector<XrActionSuggestedBinding> bindings;
-    auto add_binding = [&](XrAction action, const char* path) {
-        bindings.push_back({ action, xr_path_from_string(core_funcs_, handles.instance, path) });
-    };
+    auto add_binding = [&](XrAction action, const char* path)
+    { bindings.push_back({ action, xr_path_from_string(core_funcs_, handles.instance, path) }); };
 
     add_binding(grip_pose_action_, "/user/hand/left/input/grip/pose");
     add_binding(grip_pose_action_, "/user/hand/right/input/grip/pose");

src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp

@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 #pragma once
@@ -24,6 +24,10 @@ extern "C"
     XRAPI_ATTR XrResult XRAPI_CALL xrGetInstanceProcAddr(XrInstance instance,
                                                          const char* name,
                                                          PFN_xrVoidFunction* function);
+    XRAPI_ATTR XrResult XRAPI_CALL xrEnumerateInstanceExtensionProperties(const char* layerName,
+                                                                           uint32_t propertyCapacityInput,
+                                                                           uint32_t* propertyCountOutput,
+                                                                           XrExtensionProperties* properties);
 }
 
 namespace core

src/core/schema/python/controller_bindings.h

@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 // Python bindings for the Controller FlatBuffer schema.
@@ -119,8 +119,7 @@ inline void bind_controller(py::module& m)
                      return obj;
                  }),
              py::arg("data"), py::arg("timestamp"))
-        .def_property_readonly("data",
-                               [](const ControllerSnapshotRecordT& self) -> std::shared_ptr<ControllerSnapshotT>
+        .def_property_readonly("data", [](const ControllerSnapshotRecordT& self) -> std::shared_ptr<ControllerSnapshotT>
                                { return self.data; })
         .def_readonly("timestamp", &ControllerSnapshotRecordT::timestamp)
         .def("__repr__",
@@ -139,8 +138,7 @@ inline void bind_controller(py::module& m)
                      return obj;
                  }),
              py::arg("data"))
-        .def_property_readonly("data",
-                               [](const ControllerSnapshotTrackedT& self) -> std::shared_ptr<ControllerSnapshotT>
+        .def_property_readonly("data", [](const ControllerSnapshotTrackedT& self) -> std::shared_ptr<ControllerSnapshotT>
                                { return self.data; })
         .def("__repr__",
              [](const ControllerSnapshotTrackedT& self) {

src/core/schema/python/pedals_bindings.h

@@ -1,4 +1,4 @@
-// SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 // SPDX-License-Identifier: Apache-2.0
 
 // Python bindings for the Pedals FlatBuffer schema.
@@ -63,9 +63,9 @@ inline void bind_pedals(py::module& m)
                      return obj;
                  }),
              py::arg("data"), py::arg("timestamp"))
-        .def_property_readonly("data",
-                               [](const Generic3AxisPedalOutputRecordT& self) -> std::shared_ptr<Generic3AxisPedalOutputT>
-                               { return self.data; })
+        .def_property_readonly(
+            "data", [](const Generic3AxisPedalOutputRecordT& self) -> std::shared_ptr<Generic3AxisPedalOutputT>
+            { return self.data; })
         .def_readonly("timestamp", &Generic3AxisPedalOutputRecordT::timestamp)
         .def("__repr__",
              [](const Generic3AxisPedalOutputRecordT& self)
@@ -85,9 +85,9 @@ inline void bind_pedals(py::module& m)
                      return obj;
                  }),
              py::arg("data"))
-        .def_property_readonly("data",
-                               [](const Generic3AxisPedalOutputTrackedT& self) -> std::shared_ptr<Generic3AxisPedalOutputT>
-                               { return self.data; })
+        .def_property_readonly(
+            "data", [](const Generic3AxisPedalOutputTrackedT& self) -> std::shared_ptr<Generic3AxisPedalOutputT>
+            { return self.data; })
         .def("__repr__",
              [](const Generic3AxisPedalOutputTrackedT& self)
              {

src/plugins/manus/70-manus-hid.rules

@@ -0,0 +1,5 @@
+# HIDAPI/libusb
+SUBSYSTEMS=="usb", ATTRS{idVendor}=="3325", MODE:="0666"
+SUBSYSTEMS=="usb", ATTRS{idVendor}=="1915", ATTRS{idProduct}=="83fd", MODE:="0666"
+# HIDAPI/hidraw
+KERNEL=="hidraw*", ATTRS{idVendor}=="3325", MODE:="0666"

src/plugins/manus/CMakeLists.txt

@@ -107,4 +107,5 @@ add_subdirectory(tools)
 # Install Manus SDK shared library
 install(FILES "${_MANUS_LIB}"
     DESTINATION ${CMAKE_INSTALL_LIBDIR}
+    COMPONENT manus
 )