Megatron LoRA correctness: align distributed semantics with Megatron and validate TP/EP/ETP/DP against an oracle#619

Draft

FurtherAI wants to merge 34 commits intomainfrom

austin/megatron_lora_correctness_oracle_tests

Commits on Mar 16, 2026

Add correct data parallelism.
FurtherAI
committed

Commits on Mar 20, 2026

Commits on Mar 21, 2026

Commits on Mar 23, 2026

Commits on Mar 24, 2026

Commits on Mar 25, 2026

Create lora without instantiating full model by using meta device.
FurtherAI
committed
Update Megatron dependencies for transformers v5 change.
FurtherAI
committed
Update megatron tests for new lora kernel and avg grads across experts for stability.
FurtherAI
committed