Data conversion issues

[Gulan1]

Hello, thank you very much for your open-source project
I have a question. Assuming that I collected 500 HDF5 datasets in the first round and converted them into Lerobot for training, and then collected another 500 datasets in the second round, do I only need to convert these 500 datasets and put them together, or do I need to convert 1000 datasets together? Because dataset conversion will generate a meta file, I don't know if it will have any impact

[Tradewindycc]

Hi, thanks for using our repo!

For the data merging question:

Approach 1: Merge HDF5 first, then convert to LeRobot.
If you merge your two batches of 500 HDF5 datasets into one (1000 total) before running the conversion script, the meta statistics will be computed correctly over the entire dataset. No additional steps needed.
Approach 2: Convert separately, then merge
If you've already converted the two batches of 500 into LeRobot format independently, the meta statistics (e.g., normalization) in each batch will only reflect their respective 500 episode. In this case, you can use our LeRobot dataset merge script.
This script handles the merging process and recomputes the statistics across the merged dataset.