diffusers fails in PyTorch when generating image using stabilityai/stable-diffusion-3.5-large-turbo, black-forest-labs/FLUX.1-dev on CPU

[yurivict]

Describe the bug

Trace for stabilityai/stable-diffusion-3.5-large-turbo:

Traceback (most recent call last):
  File "/disks/samsung-4TB-A/AI-models/from-hugging-face/stable-diffusion-3.5-large-turbo/run.py", line 15, in <module>
    image = pipe(prompt).images[0]
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion_3/pipeline_stable_diffusion_3.py", line 1068, in __call__
    noise_pred = self.transformer(
                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/utils/peft_utils.py", line 315, in wrapper
    result = forward_fn(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/models/transformers/transformer_sd3.py", line 291, in forward
    hidden_states = self.pos_embed(hidden_states)  # takes care of adding positional embeddings too.
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/models/embeddings.py", line 559, in forward
    latent = self.proj(latent)
             ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 553, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 548, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^
RuntimeError: Input type (c10::Half) and bias type (float) should be the same

Trace for black-forest-labs/FLUX.1-dev:

Traceback (most recent call last):
  File "/disks/samsung-4TB-A/AI-models/from-hugging-face/FLUX.1-dev/run.py", line 14, in <module>
    image = pipe(prompt).images[0]
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 949, in __call__
    noise_pred = self.transformer(
                 ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/utils/peft_utils.py", line 315, in wrapper
    result = forward_fn(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/diffusers/models/transformers/transformer_flux.py", line 680, in forward
    hidden_states = self.x_embedder(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 134, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: mat1 and mat2 must have the same dtype, but got BFloat16 and Float

Reproduction

For stabilityai/stable-diffusion-3.5-large-turbo:

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-3.5-large-turbo", dtype=torch.bfloat16, device_map="cpu")
pipe.reset_device_map()
pipe.to("cpu")

prompt = "Bicycle in the forest, 8k"
image = pipe(prompt).images[0]

For black-forest-labs/FLUX.1-dev:

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", device_map="cpu")
pipe.reset_device_map()
pipe.to("cpu")

prompt = "A beautiful house next to the beach, 8k"
image = pipe(prompt).images[0]

Logs

See above.

System Info

OS: FreeBSD 15 STABLE amd64
diffusers-0.37.0
pytorch-2.10.0
Python-3.11

Who can help?

No response