Questionable initial load times with systemjs

[PatrickJS]

There seems to be some overhead with systemjs loader compared to others. There seems to be 800-1000ms to load angular2/angular2 lib initially even after I included the bundled version of angular. It's noticeably different from the alternatives like webpack that doesn't have the systemjs overhead since it compiles to simple callback style.
[Webpack]
[SystemJS]

[pkozlowski-opensource]

@gdi2290 it is kind of hard to see what is going on with just the info provided. What is your exact setup? Are you using jspm or just SystemJS polyfill? How do you do tranpilation / packaging?

Could you please share more details (ideally 2 relevant plunks) before we start drawing any conclusions from the screen-shoots?

[robwormald]

Had a bit of a play with this, findings:

• there's a fair bit of overhead from the System.register calls, and a fair bit more from the lookup/resolution logic. This is exacerbated with Angular2 because its got about a million modules :D
• Using the pre-bundled version of Angular2 appears to drop performance slightly - I had better results building my own bundles from source (one of the reasons I want to get angular2 up on jspm)
• Building a minified System.register bundle (from ES6 source) with all the deps (zone/reflect) comes out to around ~975kb, and takes (wholly unscientifically) ~1.5 seconds to render.

I had the best results building a minified SFX bundle - this allowed me to load the script async at the top of the head. Doing this got me to around ~800ms to render, and a big increase in perceived speed since the light DOM stuff rendered more or less instantly since it's not blocked by anything.

We should definitely set up some super-scientific benchpress specs to test this stuff out.

@guybedford might have a bit more input on this, but iirc he hasn't really been focused on performance yet over spec compatibilty.

[pkozlowski-opensource]

@robwormald hmm, I think we should clarify few things, as some of the statements are puzzling for me.

there's a fair bit of overhead from the System.register calls, and a fair bit more from the lookup/resolution logic

How did you measure this overhead? What is the baseline? What is the best "best possible" performance and what limitations does it imply? How big is the overhead-percent wise to the best theoretical base-line?

This is exacerbated with Angular2 because its got about a million modules

How come? Did you mean exports instead of modules? Still, a million would be huge - are you sure that you are not working with jspm pointing to the angular2 repo which has some dependencies on gulp-related stuff that should be moved to dev dependencies?

takes (wholly unscientifically) ~1.5 seconds to render

Once again - what is the setup? With the minimal one I've got in https://github.com/pkozlowski-opensource/ng2-play the load takes ~500 ms.

I would like to move on on this topic, but to do so we need to make sure what we are measuring / comparing here. What is the exact setup you've got (plunker would be ideal). Does it involve jspm? We need to absolutely make sure that we are not mixing transport-format performance (System.register) with the loader performance.

[pkozlowski-opensource]

BTW, first and foremost we need to chose transport format (System.register vs. CommonJS vs. requireJS vs. sth different). The loaders / package managers should be a secondary consideration at this point, IMO. So we need to make sure that the chosen format have the desired features:

• correct handling of circular dependencies
• ability to mix pre-loaded and dynamically loaded content

and performance characteristics: minimal overhead (size and time). It also needs to be widely supported and future-proof (loadable through a System.import).

[pkozlowski-opensource]

@gdi2290 @robwormald do you guys have any concrete counter-proposals for the output format (that is, replacement for System.register)? This discussion started as comparing performance of SystemJS vs. webpack and I still don't know what it means :-(

[pkozlowski-opensource]

Oh, and one last remark - please make sure that you collect any results in incognito tab with all the extensions disabled - otherwise numbers can't be trusted....

[pkozlowski-opensource]

Here is some data from my https://github.com/pkozlowski-opensource/ng2-play setup, files served locally:

We are talking about ~100ms with disabled cache. Could be faster but still it is far from announced 1.5s.

Sooooo, we really need to look into more details as to see what we are measuring / comparing here.

[robwormald]

To be clear, I wasn't complaining / inciting, I completely agree with doing this scientifically! Beyond that, I'm a huge proponent of jspm/SystemJS, to the point I'm begining to annoy people about it, so my main aim is to see how we can improve this, not throw it out.

"jspm" in this context is just a package manager and CLI - both you and I are using the SystemJS builder to do bundling (I'm using the latest/beta release).

I'm working on getting benchpress set up currently, but the basic metric was start a perf timer in a script tag, first thing in the head - and stop it before the bootstrap() call - again, absolutely not a scientific test, just an indicator of module registration being complete.

Running a CPU profile on it, its reasonably clear to me (unless I'm seriously misreading something) that there's quite a lot of work going on in the loader here - digging into the timelines shows the same sort of behavior. How much of that is overhead is what we need to work out. This is on a pretty low-power machine.

There's a huge amount of variables in play here (Traceur vs Babel vs Typescript compilation, for example), so perhaps we should get a list of scenarios down?

[robwormald]

Ah, one thing that makes a huge improvement is not importing Zone and reflect. Reflect brings in a HUGE amount of crud if imported via import syntax. Both function fine this way (i chatted with @rkirov the other day about this), but obviously they're not set up for this kind of usage yet - this is the kind of thing that's not obvious to people who aren't up on es6 modules yet ("well, you can import all of your modules like this, except for these two which are special cases....")

edit: https://github.com/robwormald/ng2-jspm-seed if you want to have a look. I'm experimenting with all kinds of combinations here - the outstanding workaround is the SystemJS bundler not yet supporting TypeScript single-file output.

[pkozlowski-opensource]

@robwormald right, so definitively, I don't think we should load zone.js / loader itself / polyfills from the package manager. Those are not real functional dependencies, but kind of infrastructure files.

both you and I are using the SystemJS builder to do bundling (I'm using the latest/beta release).

Right, so I guess boils down to the packager configuration. Do you see the same problems if you use the "official" bundle?

Beyond that, I'm a huge proponent of jspm/SystemJS, to the point I'm begining to annoy people about it, so my main aim is to see how we can improve this, not throw it out.

It is good to be fan of the technology that works for you! But if we say that we see perf problems we need to be super clear about the setup and what we are measuring. The trouble is that people use "SystemJS" to mean different things (System.* API, loader, module transport format, polyfill etc.) so we need to be precise here.

Regarding jspm - I still haven't had time to dig into it but there are things that scare the hell out of me, ex: https://github.com/robwormald/ng2-jspm-seed/blob/master/jspm.conf.js Was this file generated? Why it is so huge and lists so many things?

Now, I think that what is the most important for now is to understand if there are any issues with the System.register module format itself and if so - what are the alternatives?

From the discussion we are having so far I don't look like we've got anything tangible / actionable here....

[pkozlowski-opensource]

@robwormald I'm still not sure what you've meant by:

This is exacerbated with Angular2 because its got about a million modules :D

I suspect that this is something specific to the way you've created your bundle, but can't confirm.

One more thing - it seems like there was some work done on the perf improvements, see: ModuleLoader/es-module-loader#386

[pkozlowski-opensource]

@gdi2290 Patrick, do you have any more details / input here? For now I can't see anything actionable here, I'm afraid :-(

[PatrickJS]

Yo, I'm out of town so I haven't had time to fully read the convo. The delay is due to SystemJS. For example if you console.time System.import('app') you see about 800-1000ms. But if you put that in a setTimeout then you notice the drop dropped below 60ms. If you dig around you realize it's due to SystemJS. I wasn't comparing the two so much as saying there is a problem with SystemJS and I was using webpack, since its packaged to simple callbacks, and doesn't have the overhead of SystemJS. The other problem with SystemJS, that I didn't mention, was an IO problem since Angular has so many modules (browsers being limited to so many calls). If SystemJS allows batched modules or just HTTP2 then the IO problem could be solved

[pkozlowski-opensource]

@gdi2290 it sounds like you are comparing non-bundled version... which I believe is of limited value in practice - I mean - I wouldn't expect anyone to run with un-packaged Angular2, even for demos / quick experiments.

When you say "The delay is due to SystemJS" - what does it mean exactly? System.register format? SystemJS polyfill? If so, which one? What is the format that webpack is using as does this format cover all the functionality we need (see my earlier comments)?

Once again, we need to be more precise here and share exact setup, otherwise we are not progressing.... ping me when you are back / ready to work on this topic, but then again, I don't think we should be spending much time investigating un-packaged Angular2....

[PatrickJS]

I'm using bundled version. SystemJS no jspm

[pkozlowski-opensource]

@gdi2290 I'm totally confused now... You are saying:

The other problem with SystemJS, that I didn't mention, was an IO problem since Angular has so many modules (browsers being limited to so many calls). If SystemJS allows batched modules or just HTTP2 then the IO problem could be solved

If it would be packaged here wouldn't be any IO / HTTP calls....

SystemJS

OK, do you mean https://github.com/ModuleLoader/es6-module-loader or https://github.com/systemjs/systemjs or sth else? What version?

I think that the best is that you share your setup so I can look at it, as I don't see the delays with my minimal setup here: https://github.com/pkozlowski-opensource/ng2-play

Take your time, just ping me when you are ready to spend some time with me on it, as currently it is not clear to me what is your exact setup and I can't see the problems you are describing....

[robwormald]

@pkozlowski-opensource heh, to answer your question, yes, the jspm.conf.js file is autogenerated and managed by JSPM - think of it like npm's shrinkwrap.

I'm going to set up some benchmarks this week. One thing to keep in mind is that the SystemJS polyfill is written for a future-land where HTTP2 and the like are the norm. The theory being there that you might actually load something like angular2 unbundled, with the server effectively pushing out the rest of the bundle. How true this will actually be is unknown, but that's the theory!

@gdi2290 jspm-cli does have a --depCache parameter, which will pre-calculate the dependency tree, and load it up front, vs a bunch of roundtrips to resolve deep trees. Again, this is sort-of-theoretical, as there's very few http2 implementations in the wild.

Further reading: http://daniel.haxx.se/http2/

Guy talks about options here: https://github.com/jspm/jspm-cli/blob/9e7e7301f85c1de525e1c9636a572101c6ec3344/docs/production-workflows.md

[jeffbcross]

Yeah, like @robwormald said, the ideal scenario would embrace HTTP/2 and allow for unbundling and loading only the dependencies needed by the application (with some way to pre-load assets that aren't needed yet). The browser support is there, the server support is there, it'd be nice to complete the story with module loader support.

We should put serious thought on what the "right way" is to load angular modules, based on performance, flexibility, and ease-of-use. There are some optimizations we could do for Angular itself, like declaring angular's module-dependency graph using SystemJS' depcache. Ideally we could write some benchmarks to test this with an HTTP/2 backend.

[pkozlowski-opensource]

@jeffbcross @robwormald oh, I didn't realize that you guys are experimenting with HTTP/2 loading of un-packaged ng2 files! This should be definitively looked into / explored. But at this point it should be one of possible solutions, but not the only one. IMO we need to make sure that we don't require people to have specific server-side configuration to use ng2.

Now, the most important question for me right now is: are we taking any risks / closing any doors by going with SystemJS loader / System.register format combo? If so, what are the drawbacks? What are the alternatives?

[robwormald]

angular2 is on jspm, and we added an an override to the reflect-metadata package - there's a runtime check in the polyfill that the static analyzer was picking up and bundling in all the crypto shims.

With that fixed, doing

import 'zone.js'
import 'reflect-metadata'

doesn't massively bloat the build.

[robwormald]

@pkozlowski-opensource was talking to @guybedford, he pointed me towards: ModuleLoader/es-module-loader#386 and babel/babel#1722

This might explain some of the discrepancy we're seeing, as traceur has this optimization but babel doesn't yet.

[jeffbcross]

I created an issue to create benchmarks to measure SystemJS performance in different conditions: #2618

[jeffbcross]

I think we can close this issue, and discuss further once we have more data.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}