The Diffusers example code doesn't work, tries to get dtyye of unconditional model
Hi, I get the following error from the example diffusers code where is the branch with the optional unconditional model code ?
File "/Volumes/SSD2TB/AI/Diffusers/lib/python3.11/site-packages/diffusers/pipelines/ideogram4/pipeline_ideogram4.py", line 468, in check_inputs
raise ValueError(
ValueError: guidance_schedule must have length num_inference_steps (8), got 48
If I give it a guidance schedule with the right number of steps, or guidance scale (even if its 0.0) it tries to use the unconditional models dtype.
Traceback (most recent call last):
File "/Volumes/SSD2TB/AI/Diffusers/ideogram4_instant.py", line 50, in <module>
image = pipe(
^^^^^
File "/Volumes/SSD2TB/AI/Diffusers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Volumes/SSD2TB/AI/Diffusers/lib/python3.11/site-packages/diffusers/pipelines/ideogram4/pipeline_ideogram4.py", line 668, in __call__
neg_llm_features = neg_llm_features.to(self.unconditional_transformer.dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'dtype'
Hi, thanks for raising this. We've updated the readme and you should be able to run with the following code:
import torch
from diffusers import Ideogram4Pipeline, Ideogram4Transformer2DModel
class ZeroUnconditionalTransformer(torch.nn.Module):
def __init__(self, dtype=torch.bfloat16):
super().__init__()
self.register_buffer("_dtype_anchor", torch.empty(0, dtype=dtype), persistent=False)
@property
def dtype(self):
return self._dtype_anchor.dtype
def forward(self, *, hidden_states, **kwargs):
return (torch.zeros_like(hidden_states),)
transformer = Ideogram4Transformer2DModel.from_pretrained(
"fal/ideogram-v4-instant",
subfolder="transformer",
torch_dtype=torch.bfloat16,
)
pipe = Ideogram4Pipeline.from_pretrained(
"ideogram-ai/ideogram-4-nf4-diffusers",
revision="1874bc70267ba2c823a7239e1d70dd308c8d64dc",
transformer=transformer,
unconditional_transformer=None,
torch_dtype=torch.bfloat16,
)
pipe.register_modules(unconditional_transformer=ZeroUnconditionalTransformer())
pipe.to("cuda")
image = pipe(
prompt, # Ideogram-compatible structured JSON caption
height=1024,
width=1024,
num_inference_steps=8,
guidance_scale=1.0,
guidance_schedule=None,
mu=0.0,
std=1.75,
generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]
guidance_scale=1.0 is intentional. Diffusers computes 1.0 × conditional + 0.0 × dummy_unconditional, so no unconditional model weights are loaded and there is no effective CFG. The stand-in has zero parameters, and we verified the complete eight-step generation with the released Diffusers 0.39.0 package.
If you still encounter an issue, please let us know. We'll be happy to help.
That did the job, thank you,