Desktop memory usage

Sun Sep 10 15:43:33 CEST 2006

 Hello,

 I finally found some time to put together something that I had measured 
already quite a while ago. And I think the results are interesting. But 
before I post this somewhere publically I'd appreciate if somebody could at 
least quickly check it. I'd hate to see this trashed by people just because 
of some silly stupid mistake that I've managed to overlook. It still needs 
some final polishing but otherwise I right now consider it complete, so in 
case you see something wrong, missing or not clear with it, please tell me.

 Thanks

-- 
Lubos Lunak
KDE developer
--------------------------------------------------------------
SUSE LINUX, s.r.o.   e-mail: l.lunak at suse.cz , l.lunak at kde.org
Lihovarska 1060/12   tel: +420 284 028 972
190 00 Prague 9      fax: +420 284 028 951
Czech Republic       http//www.suse.cz
-------------- next part --------------
This was actually supposed to be a follow-up to my tests of startup performance of various desktop environments [http://www.kdedevelopers.org/node/1663 http://www.kdedevelopers.org/node/1664], primarily KDE of course :). In fact I even did most of the benchmarks some time after the startup ones, but, alas, I'm much better at writing things that computers are supposed to read than at writing things that people will read :-/ (some volunteer to write good user documentation for KWin's window specific settings, BTW ;) ?) I even meant to make a somewhat more extensive analysis of the numbers, but having never found time to write that, I decided I should publish at least a shorter variant with all the numbers and some conclusions. You can do your own analyses of the numbers if you will.

These memory benchmarks are meant to measure various cases of desktop configuration and compare KDE to some other desktop environments. Specifically, I compared against Xfce 4.2.2 (as shipped with SUSE Linux 10.0) as the so-called lightweight desktop, WindowMaker 0.92.0 as a plain window manager and GNOME. GNOME, built using GARNOME, was originally version 2.12.2, later redoing it with 2.14.0 without measuring noticeable difference in these specific cases (despite 2.14 release notes claiming performance improvements). As I no longer have the same setup I cannot redo it with the very recent 2.16 unfortunately. Simply consider this to be a bit old. The others are for comparison anyway :). KDE itself was KDE3.5.x with my performance patches, all of which are already upstream by now.

The tool used to measure memory usage was Exmap [http://www.berthels.co.uk/exmap/] - the only tool for measuring memory usage that I've ever found to be actually useful (I think I've already blogged about it ;)  [http://www.kdedevelopers.org/node/1567]). Its so-called effective memory usage numbers try to account for things like dividing shared libraries among all the processes using them, unlike tools like "top" that just report the numbers they find in /proc and nobody really knows how to interpret them. In other words, if you use things like "top" or "free" for precise measuring of memory usage, you're crazy. Nevertheless, for the crazy ones, I used also "free" alongside with Exmap, just for the fun of it, numbers from "free" will follow in parentheses. While the numbers from Exmap were stable when repeating the same benchmark, numbers from "free" vary slightly and had to be adjusted to compensate for this.

The actual measuring was roughly like this: I had SUSE Linux 10.0 running in runlevel 3, i.e. without any X started automatically. I started X manually myself, with only xterm running. Then I measured total system memory usage (total effective memory (XXX) in Exmap, -/+ buffers/cache in "free", there was never any swap usage). Launched KDE/GNOME/Xfce/WindowMaker in a particular tested setup and measured total system memory usage again, the same way. The difference of course was the memory used by the various environment itself. So, let's see the actual numbers:

Plain X:

This number is definitely worth mentioning as well. The case when no desktop is running at all, only system daemons, X and xterm. And the number is:

31.3 (43.2)

Yes, more than 30M in use already, according to Exmap (and more than 40M according to "free", but I myself trust the Exmap number more). Effective memory usage from Exmap for apps shows that 12.3M is taken by X, running at 1024x768x16, 3.0M by HAL daemon (out of which 2M is private data, don't ask me why) and xterm takes 2.2M. Since this is SUSE Linux 10.0, there isn't any ZMD or anything like that. Some of these things already running use code that is also reused by desktops, for example xterm uses some X11 libraries, so all the numbers are offset by this value, although it should be relatively small and is the same for all desktops. Although HAL daemon uses some GNOME libraries, due to using GARNOME these are different than the ones used by GNOME in all these benchmarks.

All basic tests that follow are measured against this number unless explicitly stated otherwise.

Very plain desktop:

"Very plain" means minimal functionality. In KDE's case there there was only the basic desktop and very basic panel applets like the taskbar, pager and clock. Similarly with other desktops. The desktops were fully functional though, without any "cheating" like I did during the last Akademy talk. And the numbers are:

desktop usage       diff
KDE    60.4 (57.6)  29.1 (14.4)
GNOME  74.8 (71.2)  43.5 (28)
Xfce   47.9 (52.6)  16.6 (9.4)
WMaker 36.9 (47.0)  5.6 (3.8)

I have already said the "free" numbers are these mostly only for the fun of it, haven't I? These numbers seem to differ a lot between Exmap and "free". One explanation for I have is that they interpret differently what used memory actually is. Consider files that are mmapped for read-only access, such as KSycoca or GNOME's icon caches. Exmap counts them as used memory (the parts that are really in memory), while "free" counts them as part of "cached", i.e. not considering them to be a really used memory. I personally consider the "free" approach to be wrong. In order to use KSycoca, it has to be in memory, of course, in order to read the data from it. So it really takes up some memory. Because it's a read-only mapping, this memory can be actually discarded whenever needed, so in a way it could be considered the memory is not really taken, but those data need to be loaded into memory again when they're needed and this memory is necessary for loading the contents from the file. Besides, repeated loading from disk would be of course horribly slow. If you think this is wrong thinking, just use the same logic with libraries. They are (mostly) read-only mappings of the binaries and therefore act similarly. And they indeed need to be in memory in order for applications using them to work and use them. KDE's base libraries are about 15M and if you'd prefer the "free" thinking approach, you'd have to subtract this number from KDE's memory usage. Yet of course many people complain about the fact that using any KDE application means loading KDE's libraries into memory.

As for the actual results, well, too bad. In fact I had been hoping, somewhat naively indeed, that such very stripped KDE would match Xfce like it did with the startup time [http://www.kdedevelopers.org/node/1664]. But the size of KDE's libraries is hard to beat and with such stripped desktop there is not that much that'd take advantage of it. GNOME's huge number seems to be caused by running way too many processes - even panel applets are separate processes. This will have even worse impact later.

Very plain desktop with 3 basic applications:

Desktops alone are of course not very useful. One needs to run some applications to actually make use of the computer. So in addition to the very plain desktop there were 3 applications running: A terminal, a text editor and a file browser. In KDE's case the choice is obvious, they are Konsole, KWrite and Konqueror. In GNOME's case they are GNOME Terminal, Gedit and Nautilus. The terminal button in Xfce launches XTerm, editor for Xfce is Mousepad and for file management there was XFFM. In the WindowMaker case the choice was the hardest - it is just a window manager and it doesn't come with any accompanying applications at all, besides applets. Eventually I decided on running 3 times Xterm: One as a terminal, one with "vi" as an editor and one with just shell as the "file manager". Numbers:

desktop usage       diff        very plain diff
KDE 75.3 (67.5) 44 (24.3)       14.9 (9.9)
GNOME 90.2 (83.0) 58.9 (39.8)   15.4 (11.8)
Xfce 60.1 (62.8) 28.8 (20.5)    12.2 (10.2)
WMaker 42.2 (51.0) 10.9 (7.8)   5.3 (4.0)

Not that big differences really, although generally 4-5 M per application is not that little (not though that this includes also the binaries). WindowMaker case is somewhat special in reusing the same application three times and also generally not being really desktop.

Plain desktop:

However, people usually don't use very plain desktops. Besides the basic panel applets they often run additional ones, for controlling sound volume, showing CPU usage, showing time, switching keyboard layouts and so on. So in this case the very plain desktop additionally runs some of these applets, specifically, in KDE's case, Clock, Klipper, KTimeMon, KMixApplet, KGet, KXKB, KNotes and the systray applet. I tried to make this selection in order to match the reality, but had to adjust it in order to have similar sets for other desktops (although even now it doesn't 100% match, for example I couldn't find any KGet equivalent for Xfce). The numbers are:

desktop usage       diff      very plain diff
KDE   73.8 (68.1) 42.5 (24.9)   13.4 (10.5)
GNOME 95.3 (90.6) 64.0 (47.4)   20.5 (19.4)
Xfce  48.5 (52.9) 17.2 (9.7)    0.6 (0.3)
WMaker 40.9 (49.7) 9.6 (6.5)    4.0 (2.7)

More than 10M for 8 applets isn't very impressive. The number could have been much worse or better depending on the exact sets of applets though. Some of them were real Kicker applets that are loaded into the Kicker process (Clock, Klipper, KTimeMon, KMixApplet, Systray) and as such are lightweight, others are full-blown KDE processes (and KNotes even drags in KDEPIM libraries). GNOME's bad numbers even more show that separate processes can have high costs. The Xfce value shines though. All panel applets are loaded into the Xfce panel process and are very lightweight, beating even Window Maker in this specific test (although only in the difference and in the total value). Window Maker applets are also separate processes, even if quite lightweight.

Conclusion for KDE: We should generally try to avoid unnecessary processes. There should be no need to have a full KDE application for just switching keyboard layouts and showing an icon. Unlike Kicker applets the systray even encourages this, with the practice of every systray icon being a separate applet. Even worse, most of systray icons are not even needed 99% of the time - why should KGet be there all the time when it mostly only shows an inactive icon? I've already talked to Aaron about this and the plan is to have special support for such cases that unnecessarily increase memory usage and startup time. 

Desktop with 3 basic applications:

These numbers are the sum of the two above cases. They are not necessarily real sums of the two differences though, as some code might be reused:

desktop  usage diff
KDE   85.8 (72.3) 54.5 (29.1)
GNOME 110.4 (103.0)  79.1 (59.8)
Xfce  58.3 (61.0)   27.0 (17.8)
WMaker 51.4 (59.7) 20.1 (16.5)

KDE here could do somewhat better with applets handled differently. There probably can't be done much with the big increase caused by KDE libraries, there aren't that many applications using them in this case and things like ease of development unfortunately don't show in this test. Thanks to the use of kdeinit these numbers are still noticeably lower than they would have been without it (and since prelink doesn't change that much about it, people concerned about memory usage and using prelink should try running KDE without $KDE_IS_PRELINKED set).

Xfce in these basic tests performs excellently, almost keeping up with Window Maker while actually being a desktop and not just a window manager with few applet apps and several terminals. Its integrated applets and basic applications give it an advantage here. The fact that only basic applications come with Xfce and for more advances ones like web browsers one has to go to look elsewhere doesn't show yet.

All the following tests are measured against this case unless stated otherwise.

Web browser:

A very common usage case :). In addition a web browser is launched, showing a random WWW site, specifically http://dot.kde.org ;). Browsers used were KDE's Konqueror, GNOME's Epiphany and Firefox 1.0.6. To show how applications work when not in their native desktop I measured all of them in all desktops:

desktop + browser   usage     diff
KDE + Konqueror   95.3 (83.8)  9.5 (11.5)
KDE + Firefox     108.0 (90.4) 22.2 (18.1)
KDE + Epiphany    112.5 (91.5) 26.7 (19.2) 
GNOME + Epiphany  127.7 (114.6) 17.3 (11.6)
GNOME + Firefox   133.6 (118.8) 23.2 (15.8)
GNOME + Konqueror 136.5 (113.3) 26.1 (10.3)
Xfce + Firefox    80.2 (73.2)   21.2 (12.2)
Xfce + Epiphany   82.6 (72.4)  24.3 (11.4)
Xfce + Konqueror  84.4 (71.6)  26.1 (10.6)
WMaker + Firefox  69.9 (65.9)  18.5 (6.2)
WMaker + Epiphany 74.3 (65.9)  22.9 (6.2)
WMaker + Konqueror 80.2 (65.1) 28.8 (5.4)

Here some numbers that one would expect to be the same (e.g. Firefox in various desktops) vary a bit, partially due to various reuse of libraries, partially probably due to "noise". Although the numbers are stated with precision to one decimal place, you're crazy if you don't round them up a bit more - this is benchmarking, not mathematics. The only number that seems to be somewhat strange is Firefox in Window Maker where the change is smaller than in other desktops. It may be a mistake on my side (which I can't check anymore) or perhaps some desktop-specific support in Firefox that doesn't get activated in Window Maker.

Konqueror in this case actually happens to cheat a bit, since the basic Konqueror shell is already running in another instance as a file manager from the basic setup. The basic setup usage without Konqueror running would have been 84.4, so if adjusting the numbers because of this the numbers here for the "KDE + Konqueror" case should be 1.4 higher, i.e. 96.7 usage and 10.9 diff. Its reuse of KDE technologies and the KHTML engine still make it a clear winner that's not even matched by use of Epiphany in GNOME. And, just in case that thought crosses anybody's mind, I of course made sure there were no preloaded Konqueror instances.

Interestingly enough using Epiphany in KDE needs more memory than Firefox. The same effect like with loading KDE libraries shows here, Epiphany apparently uses many GNOME libraries. It's also worth noting that Konqueror's low memory usage even manages to mostly compensate for the loading of KDE libraries.

Office suite:

This is very similar to the web browser case, this time with KWord from KOffice, Writer from OpenOffice.org and Abiword, all of them showing a simple document.

desktop + suite   usage         diff
KDE + KWord       100.0 (81.5)  14.2 (9.2)
KDE + OOWriter    166.7 (110.5) 80.9 (38.2)
KDE + Abiword     102.2 (85.4)  16.4 (13.1)
GNOME + Abiword   121.4 (117.8) 11.0 (14.8)
GNOME + OOWriter  196.3 (138.1) 85.9 (35.1)
GNOME + KWord     140.9 (115.3) 30.5 (12.3)
Xfce + OOWriter   142.1 (96.5)  83.8 (35.5)
Xfce + Abiword     77.1 (68.7)  18.8 (7.7)
Xfce + KWord       89.4 (72.3)  31.1 (11.3)
WMaker + OOWriter 124.7 (90.8)  73.3 (31.3)
WMaker + Abiword   64.8 (62.0)  13.4 (2.3)
WMaker + KWord     76.7 (66.0)  25.3 (6.3)

OpenOffice used support for both KDE and GNOME, so the numbers vary in different desktops. I originally wanted to embed a table from the matching spreadsheet counterpart, but I didn't find out how to embed a table from Gnumeric into Abiword, so I dropped that part. I don't know if it's actually even possible, but I'd expect that to increase the Abiword number by Gnumeric's about 8M because of Bonobo's out-of-process component embedding.

Numbers show that applications running in their home desktop have the smallest memory increase, as they reuse the desktop's resources. This however still doesn't completely compensate for the overhead of the libraries, although there are of course differences in functionality, so no good comparison can be made. Similarly the comparison to OpenOffice.org is not completely fair because although other office suites provide similar functionality to it they don't provide as extensive set of features as OpenOffice.org .

By the way, do you remember the complains [http://blogs.zdnet.com/Ou/?p=139] about Linux desktop being bloated, because a 128M laptop can brought down to its knees by using KDE and running OpenOffice.org? Do these numbers tell something about it?

Desktop's applications:

This one admitedly may look a lot like I'm trying to cheat. In this case KDE running KDE applications, GNOME running GNOME applications and Xfce running generic applications will be compared. Xfce will be running generic applications simply because it doesn't have any comparable on its own - no web browser, no office suite, no mail client. In KDE's case these are going to be Konqueror, KWord and Kontact, in GNOME's case Epiphany, Abiword and Evolution and Xfce will be accompanied by the most known independent applications, that is Firefox as a web browser, OpenOffice.org as office suite and Thunderbird a mail client.

And that's where the problem is. As shown above, OpenOffice.org can "outperform" pretty much everything else in these tests. However I don't have many choices if I want to rule out office suites that use either KDE or GNOME libraries in order to show my point. Let's first see the numbers (the diffs are to the state when no desktop is running at all):

desktop            usage     diff
KDE + KDE apps     143.2 (117.7) 111.9 (74.5)
GNOME + GNOME apps  174.8 (149.3) 143.5 (106.1)
Xfce + 3rd party apps 206.8 (135.7) 175.5 (92.5)

Now I guess it's obvious why this test is a bit strange. Xfce, doing really well in the first half, is suddenly by far the worst one. However, even after adjusting, it's not going to win. The difference between KDE and Xfce here is slightly more than 60M. The big effect of OpenOffice.org would have to be compensated by this value to make Xfce match KDE in this case. That would however mean OpenOffice.org almost reaching resource usage of KOffice, which seems very unlikely, no matter how you look at it - 3rd party app using its own toolkit and everything simply shouldn't be able to be as lightweight as KOffice which heavily reuses parts of KDE. So let's say that the Xfce value shouldn't be less than 150M - that still makes Xfce lose in the end. The same thing that handicaped KDE in the beginning by causing big initial usage eventually lead to lightweight applications that compensated for it.

GNOME's poor results here seem to be mainly result of having many expensive processes (out-of-process applets and so on). I vaguely remember hearing something about the possibility of running GNOME applets in-process, so GNOME developers should seriously consider doing this. However, even when ignoring this, GNOME still falls even more behind KDE in this last test. Epiphany is more demanding than Konqueror, Abiword is slightly more lightweight than KWord (although using components would most probably turn it around in favour of KOffice) and after summing up all the numbers it seems that Evolution is also more demanding than Kontact.

Conclusion for KDE: When comparing applications, KDE ones usually win. In more complicated cases like this KDE applications even more than compensate for all the KDE libraries they use and outperform pretty much all competition there is. So, whenever somebody tells us we're lame developers or that C++ is much worse language than C, we know it's nonsense. And, as far as the Window Maker case is concerned, I don't consider that to be competition. That's a different league with different users and different use cases.

Conclusion:

Ok, that's it. I tried quite hard to get these numbers and make sure they're usable. I however cannot rule any possible mistake and I'm obviously biased, so while I tried to be fair, I probably quite wasn't (however, since I myself was curious about the results, what would be the point of cheating?). So, in case you don't believe me or these numbers, you're free to redo this yourself, as long as you do your benchmarks somewhat correctly (it's really simple to do them incorrectly, trust me). In fact, since this is actually several months old, it would be nice if somebody tried with GNOME 2.16 and saved me the work.

The things we should learn from this:

- Exmap can be very useful when doing certain memory usage analyses. In comparison the usual tools like "free" are often next to useless. Some of the "free" numbers above are simply absurd - although they're included because I measured them, they're not used anywhere.

- We are never significantly worse than competition. In few cases we are slightly worse, sometimes we are about the same, but often we are better, sometimes even noticeably better. If somebody tells us that we are lame, that C++ or KDE are inefficient or similar things, they should first check with our comparable competition.

- "Desktops" like Window Maker are not comparable competition. There are large differences in feature sets and even in concepts. Their users are happy with what they have and don't care about KDE, or, in the worse case, they badmouth KDE but would never switch anyway.

- Although our libraries cause us some overhead, like the initial large requirements, they are our advantage. They allow us to write good applications that are often more lightweight than competition and of course there are many other benefits like large code reuse, many features and so on.

- However, there are still areas where we can improve. There are limits, of course, but we are not near them.