Multi-GPU Reduction & MatMul

[ashvardanian]

Current kernels are designed for a single-GPU execution. Let's scale them to multi-GPU systems. Ideally, using TMA and cooperative groups.

[maifeeulasad]

I am interested in exploring this feature for next few months. Can I get assigned to this one?

[ashvardanian]

Consider it yours, @maifeeulasad! Open a PR draft once you have something working 🤗

[maifeeulasad]

@ashvardanian I have raise a PR please review it.

[ashvardanian]

I'm mostly on a single-GPU setup the last few months, but will get back to this once I have more compute 🤗