[Kde-pim] Optimizing KMail's memory usage

[Milian Wolff]

Hello all,

I'd like to propose a change to decrease the memory consumption of KMail. 
Akonadi::ItemPrivate in kdepimlibs/akonadi/item_p.h is currently the number 
one memory consumer. As evidenced by the heaptrack [1] log I generated, see 
[2], 9191232 bytes peak-memory consumption come from Akonadi::ItemPrivate 
(search for "PEAK MEMORY CONSUMERS" in the output log). The class is pretty 
large, i.e. 216bytes on my 64 bit machine, and is created extremely often (if 
you have enough mails, that is ;-)).

# Question 1

Looking at its data members, I have an idea how to decrease the size: It seems 
as if it reserves a lot of space for *potential* changes, such as:

Item::Flags mAddedFlags;
Item::Flags mDeletedFlags;
Tag::List mAddedTags;
Tag::List mDeletedTags;

I wonder - how often is an Item actually changed? If it is rarely, as I'd 
think should be the case, we could move this data into a dynamic struct that 
is only allocated on-demand. Potentially, that could decrease the size of most 
Items by 24 bytes, approx. 10% of the total struct size. Would you think the 
added complexity is worth it? Note that according to raw numbers, the total 
memory consumption would only go down by about one megabyte, less than 1% of 
the total peak consumption...

# Question 2

Has anyone ever looked at Akonadi::ImapParser::parseQuotedString and 
...::parseParenthesizedList, and is there an alternative? Afaik Volker told me 
that it would be gone with the streams API in Akonadi or something like that? 
It's amazing how often these functions are called - they triggers hundreds of 
thousands of memory allocations which can lead to memory fragmentation and 
generally slow down the performance.

# Question 3

The memory consumption of KMime could probably be decreased by implicitly 
sharing the local part and domain in KMime::HeaderParsing::parseAddrSpec. 
Worth it?

# Question 4

Most items get a flag set of one item, containing the \SEEN flag, which wastes 
a lot of space. This could be implicitly shared (currently, the flag is 
shared, but not the QSet). Would it be OK to special-case the \SEEN flag set?

# Question 5

KMime::decodeRFC2047String is apparently decoding mailing list thread titles, 
among others. Again - this is a prime candidate for implicit sharing. The 
question again though is how to limit the cache lifetime. Afaik currently we 
leak the cache for mail adresses and domains and such. Could we maybe create 
per-collection caches and bind their lifetime to the time a collection's items 
are loaded in KMail?

# Wrap Up

I'd welcome feedback. If you want to reproduce this stuff, go download the 
sources for heaptrack and install it. It should be as simple as:

mkdir build;
cd build;
cmake -DCMAKE_BUILD_TYPE=Release ..;
make install;
heaptrack kmail
heaptrack_print heaptrack.kmail....gz | less

And yes, I'll create a proper massif-visualizer integration *eventually* ;-) 
If this stuff is really really slow and perf top shows a ton of time spent in 
__GI___dl_iterate_phdr, patch libunwind with the patch at [3] and try again.

Cheers

[1]: http://quickgit.kde.org/?p=scratch%2Fmwolff%2Fheaptrack.git
[2]: https://userpage.physik.fu-berlin.de/~milianw/kmail.mem.log
[3]: https://userpage.physik.fu-berlin.de/~milianw/libunwind.better_performance.patch

-- 
Milian Wolff
mail at milianw.de
http://milianw.de
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/