Data Preparation Failed in Kubernetes

[Syulin7]

Referencing this tutorial: https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/kubernetes.html#nemo-framework-on-kubernetes-playbook
an error occurred during the Data Preparation stage.

NeMo Container image: nvcr.io/nvidia/nemo:24.03.01.framework

ENV PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python in the container dose not work.

I fixed this issue by adding the environment variable in the mpirun command.

mpirun -x PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

NeMo-Framework-Launcher/launcher_scripts/nemo_launcher/core/k8s_templates/data_preparation/data-prep.yaml

Line 20 in d14e267

- '{{- range tuple "download" "extract" "preprocess" }} mpirun --allow-run-as-root -np {{ $config.totalProcesses }} -npernode {{ $config.procsPerNode }} -bind-to none -map-by slot --oversubscribe -x PYTHONPATH -mca pml ob1 -mca btl ^openib python3 /opt/NeMo-Megatron-Launcher/launcher_scripts/nemo_launcher/collections/dataprep_scripts/pile_dataprep/{{ . }}.py --config-path=/config --config-name=config.yaml && {{- end}} echo Data preparation complete'