Feature Request: Aule-attention

[GreenShadows]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Hardware-agnostic FlashAttention 2 implementation No compilation required. Works on any GPU.

Motivation

Is it possible to implement? The current flash attention implementation used in SD.cpp/GGML engine degrades performance on AMD terribly(Vulkan), as discussed here: leejet/stable-diffusion.cpp#1186

https://github.com/AuleTechnologies/Aule-Attention

Aule-attention provides a drop-in FlashAttention implementation that works across all major GPU vendors without requiring compilation at install time. It automatically selects the optimal backend for your hardware:

Triton: For AMD ROCm and NVIDIA CUDA (training and inference)
Vulkan: For Intel, Apple, AMD consumer GPUs, and any Vulkan-capable device (inference)
CPU: NumPy fallback for systems without GPU support.

Possible Implementation

No response

[am17an]

This is a C++ project and not a python project

[GreenShadows]

??? FlashAttention was not originally written in C++. It was first implemented in Python with CUDA kernels, not pure C++.

So rejecting Aule-attention on the grounds of “language mismatch” isn’t a fully valid argument. Especially considering that the current implementation in Vulkan does not work as it should for AMD.

[am17an]

What you're asking is to install a python package as a "drop in replacement for flash atttention" in a primarily C++ repository. We can't just do pip install

[GreenShadows]

Like any code, it can be ported to other programming languages. All it takes is willingness. Maybe someone will be interested if the topic is open.

You're the Nvidia guy. It doesn't make sense to be blocking discussion of open technologies.