framepack pipeline return pure black output when using torch.float16

[zhangvia]

Describe the bug

when using torch.float16 dtype, frame pack pipeline return pure black output. is this normal? i want to use torch.backends.cuda.matmul.allow_fp16_accumulation = True to accelerate the inference of framepack. it is a new feature of torch 2.7. but ifound that even without this feature, framepack pipeline still return black output.

Reproduction

import torch
from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel,AutoencoderKLHunyuanVideo
from diffusers.utils import export_to_video, load_image
from transformers import SiglipImageProcessor, SiglipVisionModel
# torch.backends.cuda.matmul.allow_fp16_accumulation = True
transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained(
    "lllyasviel/FramePack_F1_I2V_HY_20250503", torch_dtype=torch.float16
)
feature_extractor = SiglipImageProcessor.from_pretrained(
    "lllyasviel/flux_redux_bfl", subfolder="feature_extractor"
)
image_encoder = SiglipVisionModel.from_pretrained(
    "lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16
)
pipe = HunyuanVideoFramepackPipeline.from_pretrained(
    "HunyuanVideo-Diffusers/",
    transformer=transformer,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
    torch_dtype=torch.float16,
)

# Enable memory optimizations
pipe.enable_model_cpu_offload()
pipe.vae.enable_tiling()

image = load_image(
    "./penguin.png"
)
output = pipe(
    image=image,
    prompt="A penguin dancing in the snow",
    height=832,
    width=480,
    num_frames=91,
    num_inference_steps=30,
    guidance_scale=9.0,
    generator=torch.Generator().manual_seed(0),
    sampling_type="vanilla",
).frames[0]
export_to_video(output, "output_hunyuan.mp4", fps=30)

Logs

System Info

diffusers:0.34.0.dev0
torch:2.7.1

Who can help?

No response

[a-r-r-o-w]

It is common for models trained in bf16 to result in black outputs when run in fp16 precision due to numerical overflow issues. In this case, the transformer must be run in bf16. You could try to clip the values similar to here and see if that works (but you'll have to apply clipping at every location by finding where the models outputs become nan and not just the end of the block like done here. It varies based on the model)

Maybe this flag might be interesting to you too for bf16 case.

[zhangvia]

i test the allow_bf16_reduced_precision_reduction with the script below on rtx 4090:

import torch
# torch.backends.cuda.matmul.allow_fp16_accumulation = True
torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction = True
x = torch.randn(4096, 4096, device="cuda:0", dtype=torch.bfloat16)
y = torch.randn(4096, 4096, device="cuda:0", dtype=torch.bfloat16)
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
# warmup 
for _ in range(1000):
    out = x.mm(y)
start.record()
for _ in range(1000):
    out = x.mm(y)
end.record()
end.synchronize()
time = start.elapsed_time(end)
print(f"TFLOPs {4096*4096*4096*2/1e9/(time/1000)}")

but the result was frustrating.

precision	allow_fp16_accumulation	allow_bf16_reduced_precision_reduction	TFLOPS
fp16	True	N/A	246
fp16	False	N/A	161
bf16	N/A	True	147
bf16	N/A	False	147

it seems that the only choice is to clip every block result

[a-r-r-o-w]

Most models these days have stopped training in fp16 and are now either trained in bf16 or fp8 with possibly some layers in fp32, so the fp16 accumulation trick is probably not very useful on the more modern image/video generation models.

Can I mark this issue as resolved given the above explanation?

[zhangvia]

Most models these days have stopped training in fp16 and are now either trained in bf16 or fp8 with possibly some layers in fp32, so the fp16 accumulation trick is probably not very useful on the more modern image/video generation models.

that‘s frustrating, because The RTX 4090 GPU has a Tensor FP16 performance of up to 330 TFLOPS. However, using BF16 matrix multiplication with PyTorch falls far short of this hardware limit.

Can I mark this issue as resolved given the above explanation?

yes，thanks for your detailed explanation

[a-r-r-o-w]

Pytorch only calls into the underlying cuBLAS library implementations, and those are usually the fastest available implementations. For maximizing utilization on 4090, maybe Sage Attention and FP8 scaledmm might be better and something you can try testing