fast MHR to SMPL conversion

[AmandineBtto]

Hi,

First of all, thank you for the great work and for making the code publicly available!

I had a question regarding the fast MHR-to-SMPL conversion mentioned in the paper. I found some information in the README (specifically in the mhr2smpl folder), but I’m still a bit unclear on a couple of points:

• Is it possible to use the lightweight conversion without doing the described training procedure?
• If so, would you be able to release the missing files required for the conversion?

Additionally, I was wondering whether it would be possible to integrate this fast conversion directly into the demo pipeline. Ideally, I’d like to use the fast model and obtain SMPL parameters directly as output, without going through the full training process, and avoiding the MHR -> SMPL from the MHR repo.

Thanks again for your work, and I appreciate any guidance you can provide!

[yangtiming]

Q1: Is it possible to use the lightweight conversion without performing the full training procedure?
A1: Yes. We provide the pre-trained mhr2smpl model weights, which can be used directly without additional training.

---

Q2: If so, can you release the missing files required for the conversion?
A2: Some required files are released by the original official repository, and we are unable to redistribute them.
You only need to download SMPL_NEUTRAL.pkl and mhr2smpl_mapping.npz yourself. All other necessary components are already provided in our repository.

---

Q3: Can you provide a complete demo pipeline?
A3: Due to licensing restrictions on SMPL_NEUTRAL.pkl and mhr2smpl_mapping.npz, we are unable to provide a fully self-contained demo pipeline. We may consider to build a demo pipeline recently.
Once you download these files, you can prepare your input video and directly run Step 3: Evaluation.

---

Summary:
The only additional step required is to download SMPL_NEUTRAL.pkl and mhr2smpl_mapping.npz. After that, you can prepare your video and run the evaluation pipeline directly.

from mhr2smpl.multi_view.infer_multiview import MHR2SMPLMultiView

# Initialize (with optional SmootherMLP)
model = MHR2SMPLMultiView(
    model_path="mhr2smpl/experiments/multiview_n30000_e500/best_model.pth",
    smoother_dir="mhr2smpl/experiments/smoother_w5",  # optional
)

# Prepare views: list of (pred_vertices [18439,3], pred_cam_t [3]) 
views = [(pred_verts_cam1, cam_t_1), (pred_verts_cam2, cam_t_2)]

# Infer SMPL parameters
go, body_pose, betas, weights = model.infer(views)
# go: [3] global orient (axis-angle)
# body_pose: [63] body pose (21 joints × 3)
# betas: [10] shape coefficients
# weights: [V] per-view confidence

# Or with 3D joints (+ optional SmootherMLP denoising)
go, body_pose, betas, weights, joints = model.infer_smpl_joints(views)
# joints: [24, 3] root-relative canonical joints

Note that only single-view input is supported in the current version.

We will prepare and release a demo pipeline within 10 days after downloading SMPL_NEUTRAL.pkl and mhr2smpl_mapping.npz.

[AmandineBtto]

Thank you for the detailed answer!
I've downloaded SMPL_NEUTRAL.pkl and mhr2smpl_mapping.npz, but

model = MHR2SMPLMultiView(
    model_path="mhr2smpl/experiments/multiview_n30000_e500/best_model.pth",
    smoother_dir="mhr2smpl/experiments/smoother_w5",  # optional
)

requires mhr2smpl/data/smpl_vert_sample_indices.npy. How to obtain this file?

[yangtiming]

smpl_vert_sample_indices.npy is same as experiments//sample_idx.npy

please have a look https://github.com/yangtiming/Fast-SAM-3D-Body/tree/main/mhr2smpl#files-to-download

[AmandineBtto]

Thanks, I actually misread this part and understood it was an output from training, I'm sorry.

I copied sample_idx.npy from multiview_n30000_e500 folder. The idx in it are superior to the number of vertices (18439, 3), which creates an issue when running face_v = mhr_w[self._mhr_vert_ids] in _preprocess_view() that happens with model.infer(views). What can I do to solve that?

[yangtiming]

Could you share the full traceback of the error?

Here is the full pipeline with the corresponding files for context:

Stage 1 (SAM-3D-Body backbone)
  → MHR vertices [18439, 3] per camera view
  → Barycentric mapping → SMPL topology [6890, 3]        # mhr2smpl_mapping.npz: mhr_vert_ids [6890,3] (face indices into 18439) + baryc_coords [6890,3] (weights) → [6890,3]
  → Subsample 1500 verts + centroid-center → [4500]       # smpl_vert_sample_indices.npy: indices [1500] into [6890,3] → subsample [1500,3] → flatten [4500]
  → Multi-view fusion (MinimalMultiViewNet)                # best_model.pth: N views × [4500] → [global_orient:6, body_pose:63, betas:10, weights:N]
  → Output: global_orient[6D] + body_pose[63] + betas[10]
  → (Optional) SMPL FK → joints[24,3]                     # SMPL_NEUTRAL.pkl: (global_orient[3], body_pose[69], betas[10]) → joints [24,3] root-relative
  → (Optional) SmootherMLP temporal denoising             # smoother_w5/: window=5 frames × [76] → denoised [76] (global_orient+body_pose+betas)

[AmandineBtto]

First, I found that mhr2smpl_mapping.npz from the current MHR repo does not have a key mhr_vert_ids but only triangle_ids and baryc_center.
triangle_ids has the same expected shape, so I guess it's the same as mhr_vert_ids.

After this key modification, I got:

go, body_pose, betas, weights = model.infer(views)
                                    ^^^^^^^^^^^^^^^^^^
  File "/home/amandine/workspace/HMR/Fast-SAM-3D-Body/mhr2smpl/multi_view/infer_multiview.py", line 152, in infer
    feats  = np.stack([self._preprocess_view(v, t) for v, t in views], axis=0)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amandine/workspace/HMR/Fast-SAM-3D-Body/mhr2smpl/multi_view/infer_multiview.py", line 152, in <listcomp>
    feats  = np.stack([self._preprocess_view(v, t) for v, t in views], axis=0)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amandine/workspace/HMR/Fast-SAM-3D-Body/mhr2smpl/multi_view/infer_multiview.py", line 126, in _preprocess_view
    face_v = mhr_w[self._mhr_vert_ids]
             ~~~~~^^^^^^^^^^^^^^^^^^^^
IndexError: index 22515 is out of bounds for axis 0 with size 18439

[yangtiming]

Thank you for reporting this! The README has been updated to clarify the issue.

The official MHR mhr2smpl_mapping.npz only contains triangle_ids and baryc_coords, but our code requires an additional mhr_vert_ids field. You need follow:
https://github.com/yangtiming/Fast-SAM-3D-Body/tree/main/mhr2smpl#from-the-mhr-repository

[AmandineBtto]

Thank you for the fix!

model.infer_smpl_joints requires the file SMPL_NEUTRAL.pkl. In the README, it is stated that this file should be downloaded from the SMPL-X website (https://smpl-x.is.tue.mpg.de/). However, the SMPL-X download page only provides SMPL-X model files (e.g., SMPLX_NEUTRAL.npz/.pkl) and does not include the SMPL model files. Could you please clarify how to obtain the SMPL_NEUTRAL.pkl?

[alperari]

@AmandineBtto

I had the same question, after roaming around i noticed in ./MHR/tools/mhr_smpl_conversion/README.md they already mention this:

### SMPL/SMPLX Model Files

You'll need the official SMPL/SMPLX model files:

1. **SMPL**: Download from [SMPL website](https://smpl.is.tue.mpg.de/)
2. **SMPLX**: Download from [SMPLX website](https://smpl-x.is.tue.mpg.de/)

**Note**: If you run into issues with the `.pkl` SMPL(X) model file, try the official `.npz` files instead.

MHR/ directory should have been created if you followed setup instructions in mhr2smpl/README.md

[yangtiming]

Thank you for the fix!

model.infer_smpl_joints requires the file SMPL_NEUTRAL.pkl. In the README, it is stated that this file should be downloaded from the SMPL-X website (https://smpl-x.is.tue.mpg.de/). However, the SMPL-X download page only provides SMPL-X model files (e.g., SMPLX_NEUTRAL.npz/.pkl) and does not include the SMPL model files. Could you please clarify how to obtain the SMPL_NEUTRAL.pkl?

It should be in website (https://smpl.is.tue.mpg.de/). I will make a update.

[alperari]

@yangtiming thank you for confirming.

Eventually i ended up getting chumpy error:

/miniconda3/envs/fast_sam_3d_body/lib/python3.11/site-packages/chumpy/ch.py", line 1203, in _depends_on
    want_out = 'out' in inspect.getargspec(func).args
                        ^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?

Which was familiar to me from other projects.

There is this script https://github.com/vchoutas/smplx/tree/main/tools that cleans chumpy objects from body models.

After the removal I used this chumpy-free version of SMPL_NEUTRAL.pkl and it worked.

There is also a small note that is mentioned in MHR/tools/mhr_smpl_conversion/example.py, as follows:

def run_examples(
        self,
        input_smplx_pose_file: str,
        output_dir: str,
        smpl_model_file: str | None,
        smplx_model_file: str | None,
    ):
        if smpl_model_file is not None:
            # Unfortunately, SMPL model .pkl file may come with Chumpy, which is not compatible with
            # latest Python versions. Although the latest official SMPL model in .npz format is chumpy
            # free, the default smplx package does not support .npz file as SMPL model file.
            # So please provide either a chumpy-free SMPL model .pkl file or the official .npz file,
            # from which, a chumpy-free SMPL model .pkl file can be created.
            try:
                self.smpl_model = smplx.SMPL(
                    model_path=smpl_model_file,
                )
            except:
                print(
                    "If the provided SMPL model file is a .pkl file, please make sure it is chumpy free."
                )
                if smpl_model_file.endswith(".npz"):
                    converted_smpl_model_file = smpl_model_file.replace(
                        ".npz", "_generate_from_npz.pkl"
                    )
                # Rest of the code

So, one should keep that in mind.

[AmandineBtto]

Thanks a lot @alperari for the solution!

@yangtiming do you plan to make your converter compatible with SMPL-X? MHR model has hands and face and it would be amazing to have a fast converter.

[yangtiming]

Our mhr2smpl is developed specifically for downstream robotics tasks. We currently have no plans to train a SMPL-X version; however, I believe the training process for SMPL-X is very similar. If you need it, you can adapt the existing code to train one yourself.