The only two MIDI drivers which care about the SoundFont setting at all are CoreAudio and FluidSynth. I'm not sure which of them you are referring to.

In both cases, it seems to me that ScummVM merely tells the underlying API that "this is where the SoundFont file is; do with it as you like". Therefore, I don't think there's anything ScummVM can do about loading times, though the FluidSynth and/or CoreAudio developers might be able to.