ScummVM takes a very long time loading large sf2 files into the cache. After that, the game using this soundfont behaves as normal. The problem is just that loading into the cache when starting a game takes so long. It's even not possible to cancel the loading process.
I've found OPL-3 FM 128M.sf2 on http://www3.telus.net/anapan8/oldscardemu.htm under "Complete GM Soundfonts" which could be taken to test and look onto the matter.
The only two MIDI drivers which care about the SoundFont setting at all are CoreAudio and FluidSynth. I'm not sure which of them you are referring to.
In both cases, it seems to me that ScummVM merely tells the underlying API that "this is where the SoundFont file is; do with it as you like". Therefore, I don't think there's anything ScummVM can do about loading times, though the FluidSynth and/or CoreAudio developers might be able to.