Conflix size when load controlnet+IP-Adapter when load stable-diffusion-xl-base-1.0

[dangph-alala]

Describe the bug

It work when loading stable-diffusion-v1-5

But not work with stable-diffusion-xl-base-1.0

depth_estimator = DPTForDepthEstimation.from_pretrained("Intel/dpt-hybrid-midas").to("cuda")
feature_extractor = DPTImageProcessor.from_pretrained("Intel/dpt-hybrid-midas")

controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-depth-sdxl-1.0-small",
    variant="fp16",
    use_safetensors=True,
    torch_dtype=torch.float16,
)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)

pipeline = StableDiffusionXLControlNetImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    variant="fp16",
    use_safetensors=True,
    torch_dtype=torch.float16,
).to("cuda")

pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)

pipeline.load_ip_adapter(
  "h94/IP-Adapter",
  subfolder="sdxl_models",
  weight_name="ip-adapter-plus_sdxl_vit-h.safetensors",
)
pipeline.set_ip_adapter_scale(0.7)

Reproduction

When inference:

images = pipeline(
    prompt = "best quality, high quality",
    image=image,
    control_image=depth_image,
    ip_adapter_image=ip_adapter_image,
    strength=0.99,
    num_inference_steps=50,
    controlnet_conditioning_scale=0.5,
).images[0]

It failed

Logs

RuntimeError: mat1 and mat2 shapes cannot be multiplied (12850x1664 and 1280x1280)

System Info

• Diffusers version: 0.35.0.dev0
• Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.35
• Python version: 3.10.18
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Huggingface_hub version: 0.33.1
• Transformers version: 4.53.0
• Accelerate version: 1.8.1
• PEFT version: not installed
• Bitsandbytes version: not installed
• Safetensors version: 0.5.3
• xFormers version: not installed
• Accelerator: NVIDIA GeForce RTX 3090, 24576 MiB

Who can help?

No response