Note: Implementation uses manual DOM manipulations [sticky]

[krausest]

These implementations use direct DOM modification in the end user code. This means specific DOM updating code is written tailored for these specific tests.
Those implementation are expected to perform very close to vanillajs (which consists of manual DOM manipulations).

[ryansolid]

I believe the WASM implementations like wasm-bindgen and stdweb are in this category as well.

Worth mentioning domdiff (it's the underlying reconciler behind uhtml, lighterhtml, hyperhtml). I think stage0 is worth consideration too, but it is bit more debatable. These libraries have a reconciler atleast but there are not "bindings" just wiring up their own DOM operations in the open. Worth atleast taking a look.

[aminya]

This benchmark should not force any programming style, it should just be a framework for exercising different libraries that perform certain functionality.

If you want to consider the "effort to write the code", there are better ways to measure that. You can add "lines of code" of the implementation as a measure that shows how much code a developer will need to write to do the included functionalities.

As I mentioned in the jQuery issue, this benchmark should allow all types of libraries. People can decide to use this library or not based on the performance as well as the lines of code (a measure of effort). The personal preference for syntax or style should not be covered here.

[ryansolid]

I think lines of code are poor metric for effort. I can write very concise code that is hard to reason about yet is very performant. Or I can write Elm which is easy to write but has a disproportional amount of LoCs. That being said people can just look at the implementation and make their own judgement call assuming they do equivalent things.

The problem with non-data driven approaches is they become ultimately meaningless from a comparison perspective outside serving as a guidepost for optimal performance. The scenario presented here is an approximation of a category of the solution but shouldn't represent a specific one. Let me explain why this matters. I think there are 3 factors at play.

Having knowledge of the specific problem allows certain types of optimizations that would make no sense for a general solution. However, restricting these lower-level libraries makes no sense as there is no reason jQuery should adopt React's abstraction. It's an artificial constraint.

Conversely, every library has an escape-hatch back to the VanillasJS. So an implementor could just grab a ref and wire everything up by hand. Sure it bypasses the library but it is no more effort or complex than the VanillaJS implementation. We've seen that in implementations here. It is a tempting approach to get performance. One could argue it is even idiomatic. But any library could do the same thing.

We've gotten to a place where declarative libraries are knocking on the door of vanilla performance anyway. So it's a shrinking zone, since I'm not even sure what the idiomatic jQuery implementation looks like anymore. How Vanilla is it?

I am a little curious how jQuery performs these days myself and we have a enough reference builds they could compare against themselves, maybe it is worth a go. Like how does compare against optimal WASM written in Rust. But without some common ground these stop testing the same thing. Arguably we are past that point already but it is still of value at minimum categorizing the solutions.

[aminya]

about the lines of code

Well, LoC is one of the many things that show the effort took for writing some code. I don't mean that this is the best solution, but at least gives some context. For example, if there was some LoC number, I could see that the raw wasm-bindgen implementation is too verbose.

Maybe another way is to dd a hover effect to show the code for a certain framework.

about jQuery

That's my point. There is no reference for its performance, and because of that, people everywhere say that "jQuery is not good since it is slow!".

about the too specific optimizations

These are hard to detect, and it gets into the gray area. In my opinion, the programming style or approach is not a good way to detect this. Instead, if some implementation explicitly goes from the official API to Vanilla Js API in the human-written code, then this is a good sign for detecting this.

On the other hand, there may be some frameworks that have made these optimizations their "official API". For example, they say do this if you want to achieve this functionality. I think we should allow these types of implementations as long as they are considered "first-class API".

This is not hard to imagine if finally, WebAssembly gets direct access to Web IDL APIs. This means wasm based frameworks can beat JavaScript. If this happens, we should start porting our JavaScript libraries to Wasm. Using AssemblyScript, this will not be that hard! Maybe we get a "solid-wasm" someday!

[leeoniya]

a jQuery impl would be pretty much identical to vanilla, both in perf and imperative nature.

jQuery is "slow" because people tend to over-use selectors in loops (and selectors were in software/Sizzle, not querySelectorAll). jquery code frequently manipulates attrs rather than properties, and use things like innerHTML and string parsing for creating DOM elements via e.g. $("<div>").

if you avoid using these jQuery facilities, you just end up with a super thin sugar layer over vanilla dom, and it all becomes the same...and pointless.

btw, https://github.com/franciscop/umbrella is a modern/tiny jQuery alternative, with a matching API.

[aminya]

That's why I am interested to see (one or more) implementations here. When there is no point of comparison, we can't compare them.

Here it points to different jQuery like libraries. It would be nice to implement these compare them against each other.
https://github.com/franciscop/umbrella#alternatives

We can also have a jQuery 4 implementation which is interesting to look at.

[StefansArya]

@aminya
I love the idea about "lines of code" for framework benchmark implementation. I begin to imagine if there also a preview of the code on how the framework implement the test cases, instead of diving thru test's source code.

Well jQuery is a library..
For me it's different with a framework that have template system, two-way data binding, SSR, and architecture like MVW, MVVM, or MVC. While framework give many facility for the developer to build a whole application, library give facility to help developer with a specific part. Some of the framework need to compile the app's source code separately and some of them can compile it on the fly, while library just like plug and play. Usually framework that bring the compiler on the fly will be fatter in size and need more time to boot up on the browser. Library can depend with other library and developer can use it for his app just like adding one line of code. But when a library depend on a framework, developer may can't use that library on other framework.

Some people may don't know how to benchmark JavaScript, but actually if you want to compare library performance you can easily use jsbench.me to do that. Based on your mentioned link I created a simple benchmark for benchmarking jQuery and it's friends. I included ScarletsFrame on the test as a sample framework because it has jQuery like feature.

---

Why jQuery is not included? it's because the main goal of this project is about framework benchmark.
Let's try about the framework template system and how we can compare it with jQuery.
Some random Framework:

<!-- The framework may need to compile/parse this template on the fly -->
<div id="content">
    <div @for="val in list">dummy {{ val.text }}</div>
</div>

// Depend on framework design, the implementation or the available feature
// may be different with other framework
var content = new Framework('#content', {
    list: [{text: "Henlo"}]
});

// Data-driven framework may have feature that doesn't
// need you to update the DOM manually

// Editing first array value
content.list[0].text = "Hello"; // Framework may already bind this property and update the DOM

// Add new data into the array
content.list.push({text: "World!"}); // Framework may immediately add new element into the DOM

---

With jQuery: example result

<div id="content"></div>

var content = $('#content');
var list = [{text: "Henlo"}];

// We need to manually update the DOM to match the list
for(var i=0; i<list.length; i++){
    // Bottleneck: jQuery need to parse the string to generate new element
    content.append(`<div>dummy ${ list[i].text }</div>`);
}

// Editing first array value
list[0].text = "Hello"; // We need to manually update the DOM after change this
content.children().eq(0).text(`dummy ${ list[0].text }`);

// Add new data into the array
list.push({text: "World"}); // Again.. we need manually update the DOM
content.append(`<div>dummy ${ list[1].text }</div>`);

For me not including jQuery on the test is acceptable, because the level of it's implementation was different as a framework. It's not like I hate jQuery, I love with their simple syntax for some specific case. However framework is more optimized to handle multiple case in elegant way.

Maybe we get a "solid-wasm" someday!

Calm down bruh.. Don't lit the war, or I will join the forces..

[krausest]

To be honest: I'm not too excited maintaining a few jquery based (or similar) frameworks.
I think the biggest issue is that it's hardly possible to define rules for a programming style such that it doesn't converge to vanillajs.

[aminya]

Based on your mentioned link I created a simple benchmark for benchmarking jQuery and it's friends. I included ScarletsFrame on the test as a sample framework because it has jQuery like feature.

These types of benchmarks are far from real-world operations (who wants to calculate the length!). In contrast, the benchmarks in this repository resemble the real-world situations quite well.

[StefansArya]

@aminya It's true that the benchmark I provided is about counting the available element that have dummy class inside of 2 div element. But do you think you will never want to count the available elements that match with the selector's rule?

The test code on JSBench is editable, you can just modify it into .text('my test') instead of .length.
It's also true that these type of benchmark is the small part of the real world operation, but it's doesn't mean these small part will never affect the bigger parts! -- who wants to calculate the length?

A library does provide some collection of the small parts or tools which help developer build their own design or architecture. But with a framework these small parts is managed by the framework itself to provide the medium parts so the developer can immediately design their app without the need of managing the DOM directly. The benchmark on this repository is about to compare performance on how the framework handle these small or native function of JavaScript with some given situation. We may need to manage or accessing the DOM directly if using jQuery just like VanillaJS does. Instead of that we may need to use query selector more often or parsing text for creating an element to avoid it being very dependent with native DOM functions on the benchmark's source code.

[ryansolid]

@krausest Do you agree that the WASM libraries probably should be marked as reference build due to this issue? I think there is more grey area elsewhere but like wasm-bindgen and like stdweb are 100% direct dom manipulating reference builds.

Due to some older libraries being removed due to inactivity I think that the number of implementations using imperative methods for select row has reduced. I'm onboard tightening up the definition of data-driven a bit here although I suspect at this point there might be enough libraries in the greyer area that it could almost fill its own chart. Ignoring explicit/implicit event delegation which I think is hard to argue because it can be the "the way" to do stuff in some libraries.

I've tried my best to look at every implementation(except reflex-dom) and categorize it fairly. While I like to think I understand how every library works I can make mistakes so bear with me.

What I'm calling dirty model is less obvious a problem as it works without direct DOM manipulation. It just a bit localized since it means if this was say a global store you'd carry over the selected state to every different view. It goes against the universality of the solution. But on the otherhand isn't unreasonable, although it sort of turns select into another partial update test.

Also the difference between say the end user assigning (calling a setter) to an element directly in these cases or a proxy is a mute point. You may have internalized the mechanism but the end user still needs this imperative break out. It still syntactually is a per line mutation to the underlying data model instead of a hoisted selected state lookup. Some libraries have the ability to have the store still only have the single selected row but pass escape hatches to the user in the update cycle to write directly (Solid used to do this). This is all the same thing as it gets away from declarative binding syntax in your template.

For this reason I'm more comfortable categorizing a library like attodom or redom so. It's hyperscript returns dom elements so there is no abstraction there. But setting the class name assignment not in the HyperScript but in a side mutation is clearly imperative escape hatch. Sticking to this criteria of nested selection assignment in user land I can fairly easily categorize all the implementations. But I could see how that is idiomatic attodom/redom.

Clean

solid (after #794)
domc
solid-state (after #794)
ko-jsx (after #794)
vuerx-jsx (after #794)
mobx-jsx (after #794)
ivi
petit-dom
inferno
dominator
domvm
uhtml
hyperapp
imba
endorphin
lighterhtml
elm
hyperhtml
svelte
bobril
dyo
lit-element
resonatejs
aurelia
dojo
reaml-preact
angular-optimized
crank
hullo
datum
riot
isotope
vue-next
preact
neverland
heresy
binding-scala
misojs
mithril
angular-noopzone
vue
nervjs
react-redux-hooks
yew
vidom
react-hooks
reaml-react
glasgow
angular
react-tracked
vue2-composition-api
helix
angular-ng
react-easy-state
marko
ganic
angularjs
maquette
react
knockout
react-redux
rax
ractive
react-mobX
reason-react
reagent
miso
choo
blazor-wasm

Dirty Model (put selected on each row data)

mikado
sinuous
scarletsframe
sifrr
crui
san
mimbl
lit-html
doz
ember
glimmer

Direct DOM Manipulation

vanillajs1
vanillajs
stage0
fidan
vanillajs-wc
wasm-bindgen
fullweb-helpers
fullweb-template
domdiff
stdweb
attodom
redom
marionette
fntags
apprun
marionette-jquery

---

Reflections:
On the positive there are less non-reference direct dom libraries than expecting. Some libraries clearly depend on using vanilla js to operate or like marionette is just orchestrating whatever you put in. But there are libraries like apprun that look completely declarative and then sneak direct DOM updates right on the last couple lines, and Fidan is of similar nature. Attodom, redom, and fntags are all very similar in that they generate the DOM via templates or HyperScript but then use manual DOM manipulation to bind expressions and perform updates.

Some popular libraries are dirtying the model: lit-html, ember. lit-html is especially interesting as the lit-element does not. Given the lit-element version doesn't do WC per row I imagine the majority of the overhead between these 2 approaches might be something like this.

The vast majority of libraries using direct dom and dirty models are in the top 30 but not all of them. The top 30 looks very different if this is corrected.

[adamhaile]

Counter-proposal: instead of trying to define the impossible (and inflammatory), like what's a hack / dirty / cheating etc, solve the problem from the other side by reporting results to only 1 significant digit in the results app. Aka all these frameworks are at 1.0, these at 1.1, these at 1.2 etc. No 1.06, 1.07 etc, which is meaningless precision anyway.

Randomize the sort order of results that are equal to that precision.

So from the m85 results, vanilla, mikado, solid, stage0, sinuous, fidan and domc are all in the 1.0 club and a random one will happen to be "first" each time the page reloads. Congrats guys, time to focus on something other than the last 1% or, even worse, the hope that you get a "lucky run" and show up at the top of the next results page, or even worse than that, whether another dev's 1% improvement was cheating or not.

For framework developers, have an internal flag in the results app to switch it into "meaningless mode" where 3 significant digits are used instead of 1. At least, that's how I used to run it when I was comparing two approaches in Surplus.

I think this would maximize the really awesome parts of this benchmark -- devs using it to improve their frameworks and cross-pollinate ideas -- while minimizing the really sucky parts -- devs trying to stick stink tags on their competitors.

[aminya]

@adamhaile Your suggestion seems out of the scope of this issue. You should probably create another issue for that, so it is not lost here.