Fix img2img speed with LMS-Discrete Scheduler

[NotNANtoN]

Casting self.sigmas into a different dtype (the one of original_samples) is not advisable. In my img2img pipeline this leads to a long running time in the integrate.quad call later on- by long I mean more than 10x slower.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

@anton-l could you take a look here?

[anton-l]

@NotNANtoN which dtype is causing the slowdown for you? float16?

[NotNANtoN]

@anton-l I'm not sure, to be honest, I think it is fp16. The weird thing is that it runs and returns the correct output but is incredibly slow. No idea how this can happen. But independent of the type, I think casting the sigmas in place in that function does not make sense regardless.

[patrickvonplaten]

@anton-l could you take another look?

@NotNANtoN we'll need a reproducible code snippet here though - could you post a couple of lines that showcase the slow down? I highly doubt that this PR will make the full stable diffusion pipeline 10x faster - most of the time is spent either way in the model's forward pass.

[anton-l]

Ok, tested more in-depth! The proposed changes don't affect the fp32 inference, BUT for fp16:

Without the PR:

src/diffusers/schedulers/scheduling_lms_discrete.py:149: IntegrationWarning: The maximum number of subdivisions (50) has been achieved.
  If increasing the limit yields no improvement it is advised to analyze 
  the integrand in order to determine the difficulties.  If the position of a 
  local difficulty can be determined (singularity, discontinuity) one will 
  probably gain from splitting up the interval and calling the integrator 
  on the subranges.  Perhaps a special-purpose integrator should be used.
  integrated_coeff = integrate.quad(lms_derivative, self.sigmas[t], self.sigmas[t + 1], epsrel=1e-4)[0]
100%|█████| 40/40 [00:40<00:00,  1.02s/it]

With the PR:

100%|█████| 40/40 [00:04<00:00,  8.83it/s]

So we get a 10x speedup by making sure that scipy doesn't bug out 😅

Thank you for catching this @NotNANtoN! I'll make sure that the PR doesn't break any integration tests and merge tomorrow :)

[NotNANtoN]

Thanks, @anton-l for replicating this issue! I could not get to the bottom of what caused the slow down so I was beginning to question whether it was also an issue for other people.