Review Request: Fixing massive memory consumption in KWin using the raster graphicssystem.

[Aaron J. Seigo]

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://git.reviewboard.kde.org/r/101385/#review3402
-----------------------------------------------------------


those are some impressive results. my guess is that we likely have a very similar issue in plasma. in both cases, we are dealing with lots of small bits of memory in the form of pixmaps, strings, etc. that are fairly constantly coming and going. switching between compositing and no compositing is probably the 'worst case' real world scenario for this.

if this does test out well, i'd suggest putting this code in libkworkspace so that all of our apps with this usage pattern (plasma-desktop, plasma-netbook, krunner, kwin, ..) can take advantage of a common implementation. a set of static methods in the KWorkspace namespace should suffice.

if you could repeat your experiment with plasma-desktop with and without such a patch, that would be great and let us know whether we should adopt such a strategy across the plasma shell components.

as for portability ... i'm not overly concerned about the portability of this to non-glibc systems as those systems are our primary targets. however, instead of checking for GLIBC the code could simply check for M_TRIM_THRESHOLD. for comparison, here is how busybox does this same trick for both the trim size and mmap request size threshold:

#ifndef PAGE_SIZE
# define PAGE_SIZE (4*1024) /* guess */
#endif
#ifdef M_TRIM_THRESHOLD
    /* M_TRIM_THRESHOLD is the maximum amount of freed top-most memory
    * to keep before releasing to the OS
    * Default is way too big: 256k
    */
   mallopt(M_TRIM_THRESHOLD, 2 * PAGE_SIZE);
#endif
#ifdef M_MMAP_THRESHOLD
   /* M_MMAP_THRESHOLD is the request size threshold for using mmap()
    * Default is too big: 256k
    */
   mallopt(M_MMAP_THRESHOLD, 8 * PAGE_SIZE - 256);
#endif

i doubt we need to tweak the mmap strategy, but i like the checks for the definitions of mallopt there. for portability to non-unix-y systems, we could wrap this whole thing in a #ifndef Q_WS_WIN block which should keep the KDE Windows team from pulling their hair out over the use of such UNIX-y API ;)

- Aaron J.


On May 18, 2011, 8:53 p.m., Philipp Knechtges wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://git.reviewboard.kde.org/r/101385/
> -----------------------------------------------------------
> 
> (Updated May 18, 2011, 8:53 p.m.)
> 
> 
> Review request for kwin and Plasma.
> 
> 
> Summary
> -------
> 
> The intention of this patch is to lower the heap fragmentation in KWin when using the raster backend.
> 
> One can illustrate the issue with the following testcase: If one currently uses the raster backend and
> switches back to non-compositing mode one gets easily a similar memory usage like the following:
> 
> Situation: 14 windows, QtCurve
> KWin started with compositing: 40 MB
> KWin switched to non-compositing : more than 70 MB
> 
> The first guess might be, that this is due to a memleak because of not properly released pixmaps.
> But calling malloc_stats() sheds some more light on the subject (non-compositing mode):
> 
> Arena 0:
> system bytes     =   72232960
> in use bytes     =    8180512
> Arena 1:
> system bytes     =     135168
> in use bytes     =       4784
> Total (incl. mmap):
> system bytes     =   73080832
> in use bytes     =    8898000
> max mmap regions =         13
> max mmap bytes   =   36343808
> 
> In other words, the glibc has allocated more than 70 MB memory although KWin uses only less than 10 MB.
> The problem is that glibc only resizes the heap if the heap has more than 128 KB of free space at the end, but
> many decoration pixmaps are smaller. Using mallopt to tune the threshold to 20 KB (i'm open for other suggestions?)
> fixes the issue. After the patch:
> 
> KWin in compositing mode: 19 MB
> KWin switched to non-compositing: 13 MB
> 
> 
> Arena 0:
> system bytes     =   12374016
> in use bytes     =    6894544
> Arena 1:
> system bytes     =     135168
> in use bytes     =       4784
> Total (incl. mmap):
> system bytes     =   13750272
> in use bytes     =    8140416
> max mmap regions =         67
> max mmap bytes   =   99463168
> 
> Are there some negative effects?
> The only negative effect i am aware of is, that Glibc free() is calling the sbrk syscall more often. But this should
> not be a bottleneck.
> 
> 
> Diffs
> -----
> 
>   kwin/composite.cpp 9edb99d 
>   kwin/main.cpp f767f54 
> 
> Diff: http://git.reviewboard.kde.org/r/101385/diff
> 
> 
> Testing
> -------
> 
> Tested using a standard Fedora 14.
> Would be nice to know, whether other OSes have similar issues?
> Martin Gräßlin had some concerns about the portability?
> 
> 
> Thanks,
> 
> Philipp
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.kde.org/pipermail/plasma-devel/attachments/20110519/4d6ebd1a/attachment.htm