Saturday, May 24, 2008

Javascript core performance in actual browsers

Today more and more sites do use JavaScript excessively. Be it because of Ajax-functionality or CSS/JavaScript hacks, disabling JavaScript does not work on a daily based surfing any more. On the other side all major browsers got a new version released in the last two month or do have beta versions in the queue. Time to sum up some Javascript core performance tests to show which browser vendors/developers did their homework (right).

webkit.org's sunspider performance test


Result of Javascript performance tests in short

The graph above shows the time needed by every browser for the webkit test suite (running in a loop, time in milliseconds):

FF 2.0.0.14 FF3 RC1 IE 7.0.5730.13 Opera 9.27 Safari 3.1.1
26271,4 5447,0 40846,2 12358,2 6318,8
Time in milliseconds the browsers needed in this performance test

See the details of every webkit test in the graph below.

As stated the graph above shows the overall results of webkit.org's sunspider performance test.
webkit.org is an open source project which provides the source code the Safari browser (from Apple) and Adobe Air (from, well... Adobe) are based on. It hosts an online Javascript performance test under http://webkit.org/perf/sunspider-0.9/sunspider-driver.html (caution, this link automatically starts the test).

Why "yet another Javascript test"?

In this test the five browsers "Firefox 2.0.0.14", "Firefox 3 RC1", "Internet Explorer 7.0.5730.13", "Opera 9.27" and "Safari 3.1.1 (on Windows)" were tested, technically spoken pretty simple because: Firefox 3 RC1 was released some week ago along with the XP Service Pack 3 (system and/or Internet Explorer 7) updates and a valid performance comparison should be done with every major update.

Browser usage of goit-postal.blogspot.com: Remember, it's a website for technicians

So this test shall give an expression about the core performance of Javascript in yet available mainstream-browsers (and no, thank you, no flame wars intended). That is the reason why, for example, Opera 9.27 as last stable was tested and not the faster Opera 9.5beta (according to this place) and IE7 despite of IE8beta, so called nightly builds of the browsers were abandoned for the same reason. Compatibility and conformity is not an issue here, either.

That, too, is the reason the sunspider performance test was chosen for this benchmark: Sunspider mostly tempers with core JavaScript functionality, namely some mathematics or date calculations and string manipulations. That is why it is highly limited in its characteristics i.e. for "graphical" Ajax websites using DHTML and manipulating DOM (see "Limitations of the test" below). On the other hand this performance tests nature serves to test the base performance of the Javascript engines (i.e. low-level scripting speed and the speed of the interpreter / number crunching) from the mentioned browsers.

The performance test in detail

The test was done on the following machine/circumstances:
  • Intel PC with Pentium D 3.4GHz Dual Core (so environmental task do not play such a huge role becaus they may run on the other core), 2GB RAM (just used 380M of them), GeForce8600GT (this too means the graphics system did not really interfere that much ;-)
  • Windows XP SP 3 (SP3 freshly installed)
  • Internet Explorer Version 7 (patched through SP 3)
  • Forefox 2.0.0.14 and (more interresting) Firefox 3 Release Candidate 1
  • Opera 9.27
  • Safari 3.1.1 (Windows edition, of course ;-)
  • The browser was the only running application (despite the TaskManager)
  • All unnecessary services of Windows XP (or other vendors than MS) were stopped
  • The caches of the browsers where empty, the internet connection was dedicated (for the test)
  • All browsers were opened with the same resolution (1280x1024 on a 1900x1200 panel)
  • Browser plugins were not installed or, if so, disabled (if you installed Firebug on Firefox remember to disable it for testing purposes!)
At the end of this article you will find links which help to compare the results with your own computer/environment.

The webkit-tests themselves split into "3d", "access", "bitops", "controlflow", "crypto", "date", "math", "regexp" and "strings". Below is a graph showing these (without "strings" for better visualisation).

Single performance tests on webkit.org the browsers needed to complete

The graph above shows the time needed by the browsers to complete the tests (in milliseconds):


FF 2.0.0.14 FF3 RC1 IE .0.5730.13 Opera 9.27 Safari 3.1.1
3d 3.596,6 722,0 1.890,6 1.218,4 753,0
access 2.531,8 827,4 2.489,6 1.868,2 981,2
bitops 4.668,2 641,0 2.152,8 1.750,4 762,6
controlflow 225,2 75,8 663,0 265,6 171,8
crypto 1.505,8 387,8 1.381,2 593,0 497,4
date 4.859,4 481,8 1.266,0 619,0 668,8
math 2.643,8 652,4 1.697,0 1.312,8 775,6
regexp 1.115,8 327,4 406,2 0,0 362,4
string 5.124,8 1.331,4 28.899,8 4.730,8 1.346,0
Detailed time in milliseconds the browsers needed to perform the tests

The "string" - test is an exception in two ways: Firstly it is not visualized above because it would spoil the graph (the bar of the IE would be too long making other results more or less unreadable) and secondly it shows that the Internet Explorer at least has problems with strings and, probably, memory management (depending on strings and the testing-code).

Noteable, too, is the fact that the Opera-test on "regexp" delivers a timeline of zero milliseconds. Assumable there is a Javascript-error in Opera preventing the counter to count the time correctly.

Memory and CPU usage on webkit's test

One word about the memory: The memory was released after every test (webkit runs the tests more than once) by every browser as expected, because the tests are loaded separately in frames. All browsers (despite FF2 who nevertheless did aparently well in this test) do release memory of non-needed frames, nowadays.

Another
word about the CPU-usage: This is not totally empirical but I measured CPU usage from the lowest to the max utilization of usage like this:
  1. Safari
  2. Firefox3
  3. Opera
  4. Firefox 2
  5. Internet Explorer 7
I had no good tool on Windows XP to take an absolute accurate measure of the CPU usage but could survey the usage with a process messaging tool plus my ears: The computer I sat next to and tested this really sounds bad if an application needs only a little bit more performance than others. I definitely should buy another cooler for that processor if I want to keep it ;-).

Nevertheless, noteably is that only the Internet Explorer utilized (and needed) more than 50% of the overall CPU performance on the system (up to 80%): Seems that it is the only system that can utilize more than one CPU (and needs it, anyway ;-). But to be honest, Firefox 2 didn't sound good, either... .

Limitations of the test

Never forget that this is no "real-life-performance-testing". The tests driven here are, to say the least, mostly academical. At least they are not realistic as long as someone wants to write a ray-tracer in a scripting language. But they are intended (and pretty accurate) to give a feeling of the Javascript engines working in the tested browsers.

Additionally, the test was done only regarding "stable" versions of the tested browsers in the given environment. A well-used real-life-existing test like webkit served well because my interests are real life scenarios. I am sure someone can pimp up the string-tests of webkit.org for IE or install another version of Opera (not to mention the performance of Safari running native on Mac OS X, who knows), but that is not the essence of this test, that would be something different (like a performance tuning test). This test only shows the direction.

This means (of course): This test says (nearly) nothing about real-life-performance of, let's say, heavily used Ajax-features. It does in no way measure the speed how DOM/DHTML or even graphical rendering is done by the browsers. Or garbage collection of Javascript in general (perhaps in special :), and therefore, in the real browser world. This would be a complete other kind of testing (Upcoming soon! Watch out your theatre or this site for more details!).

Conclusion: Why this test has been done and what I learned from it

Of course, testing done right needs time. Preparing the machine, restarting, emptying the cache, providing in any way you can think of that the data is as valid and correct as possible. Why did I waste my time although there are so many Javascript-testing-scenarious and results out there? Because the world keeps on changing. If you only change one single component (in computer life that normally is a new browser or
[minor] operating system version), all other components (like speed, memory etc.) may behave in an other kind.

One thing I learned (and was more than eager to test) was that Firefox 3 is using a new memory management, system wide (and for Javascript). I know that the old memory management was not at it's best, not to speak about Javascript memory leaks. Not the best base to throw in a long running RIA application. Another thing was that I heard that FF 3 competes better than Safari on the webtoolkit - test.

webkit.org provides the source code for the Safari browser. I tested some very complex Ajax framework on a Safari browser and was astonished about it's speed. Some day I heard that Firefox 3 slew Safari on it's own battleground: The webkit sunspider performance test.
Time to figure that out.

Nevertheless, (as you surely guessed till now) providing a so called "RIA" Application based on Javascript with my colleagues, it was time to make detailed and own-experienced performance testings for the existing possibilities out there. The surprising parts (some seen at the charts above) are, more or less that
  1. Webkit (Apple) astounded me because our (JS/Ajax-)applications ran recognizable three to four times faster using Safari on Mac, I saw this month ago (and though compatibility issues are a problem we really think to deploy this platform)
  2. Firefox 2 was beginning to be not as slow as Firefox 1.0/1.5 (or Mozilla, but it is still sometimes too slow) and then started to get slower again (memory usage "degregaded")
  3. Firefox 3 came with a new memory management and the capability to also resolve memory of Javascript-objects of certain depths (an old known Firefox 2 bug), so Firefox 3 definitely was/is an option
  4. The crown of performance is not the Internet Explorer any more (webkit, anyone?)
  5. Safari scared the hell out of me being that performant in ("graphical") Javascript (DHTML/DOM/Javascript-performance)
  6. I read that Firefox 3 should be faster than Safari at webkit.org which - frankly - I could not believe (and did not test on the longtrail. But I will.)
What is missing

Testing the core performance of Javascript in todays browsers is okay. But it does not give real facts or benchmark results you may rely on deploying complex Javascript-based Rich Internet Applications. Even so, every application is different.

The next performance test on Javascript will try to clearify DOM/DHTML - changes and the performance-behaviour respective to the used browsers and browser rendering engines. My guess at the moment is that webkit (Safari) is top-notch there, but who am I to say that in a changing world?


Appendix: Performance-Links webtolkit.org (alphabetically):

Performance of Firefox 2.0.0.14
Performance of Firefox 3 RC1
Performance of Internet Explorer 7.0.5730.13
Performance of Opera 9.27
Performance of Safari 3.1.1




Labels: , , , , , , , , ,

 

4 Comments:

Anonymous Oliver said...

WebKit.org has a discussion of how and why the test was chosen at http://webkit.org/blog/152/announcing-sunspider-09/

It does do a reasonable amount of AJAX-y things (the regex and string tests are largely things that ajax librariesdo -- all those JS compression libraries that actually have to decompress data are entirely limited by JS math and string performance). The other test are certainly getting closer to academia, but still actually happen in real world sites -- take John Resig's Processing.js, etc.

The other thing to realise is that we created SunSpider as a way to performance test javascript so we could find places where we were performing slowly (or slower than was strictly necessary).

Finally a word on IE. We're aware of a number of bugs in the SunSpider suite (that will be fixed), the most critical being in the string tests. SunSpider accidentally uses 'SomeString += SomeOtherString' which is actually not a standard JavaScript feature, and just happens to be supported by every browser other than IE, so while no one has actually worked out what IE is doing there it is having to do something else which is possibly a lot more expensive.

7:11 PM  
Blogger Georgi said...

Hello Oliver,

that is, what I call, valuable input. Of course I wrote the (rather long) article with something in mind: to learn from user input.

Of course the webkit is a fantastic way to learn about Javascript performance in principle, but it is limited in two ways. The first is not bad at all, namely that webkit is not optimized for browsers. Example here is IE. By the way, did you ever try to replace 'SomeString += SomeOtherString' through 'var SomeString += SomeOtherString'? Just a sophisticated guess, I am not a Javascript-developer... ;-)

The second thing I would critizise on webkit is that it is not running long enough. We wrote an Ajax application that has to function for four to eight hours in the browser without restarting it or even reloading the page. Keeping that in mind I am sure you agree that this kind of applications need other testing tools as webkit in general.

We solved the problem but only for Firefox (and perhaps Safari, there is not enough data to tell now). We know that we definitely did not solve that for IE, because fragments in DOM rest there (not cleaned up from us now).

Do webkit is - in my opinion - what it is, a great tool to test core Javascript speed.

Hope that helps anyway, Georgi

7:27 PM  
Anonymous Oliver said...

If you mean by "do webkit" you mean the webkti sunspider test then i agree -- but coming up with a real world example of DOM usage that is sufficiently long running and cpu bound is difficult -- most turn into microbenchmarks (eg. how fast an you add an element to the DOM element becomes "for(var i = 0; i < 100000; i++) addElement")

And WebKit is the fastest engine in most of the microbenchmarks i've seen.

As far as changing SunSpider goes, because it is used so widely now making changes (even corrections) is difficult, because it invalidates (effectively) any existing results that people have published.

Curiously you said you had compatibility issues with webkit -- have you filed them at http://bugs.webkit.org ? A bug can't be fixed if we don't know about it :D

--Oliver

9:43 PM  
Blogger Uzbekjon said...

Great write up.

For those of you who is interested in your own javascript code performance testing you can find a nice article here:

http://jquery-howto.blogspot.com/2008/12/how-to-test-javascript-performance.html

11:42 PM  

Post a Comment

Links to this post:

Create a Link

<< Home