[Kde-pim] Optimizing KMail's memory usage

[Daniel Vratil]

On Monday 25 of August 2014 20:16:25 Milian Wolff wrote:
> Hello all,

Hi Millian,

first - thanks a lot for looking into this and suggesting improvements - we 
need this a lot :-)

> 
> I'd like to propose a change to decrease the memory consumption of KMail.
> Akonadi::ItemPrivate in kdepimlibs/akonadi/item_p.h is currently the number
> one memory consumer. As evidenced by the heaptrack [1] log I generated, see
> [2], 9191232 bytes peak-memory consumption come from Akonadi::ItemPrivate
> (search for "PEAK MEMORY CONSUMERS" in the output log). The class is pretty
> large, i.e. 216bytes on my 64 bit machine, and is created extremely often
> (if you have enough mails, that is ;-)).
> 
> # Question 1
> 
> Looking at its data members, I have an idea how to decrease the size: It
> seems as if it reserves a lot of space for *potential* changes, such as:
> 
> Item::Flags mAddedFlags;
> Item::Flags mDeletedFlags;
> Tag::List mAddedTags;
> Tag::List mDeletedTags;
> 
> I wonder - how often is an Item actually changed? If it is rarely, as I'd
> think should be the case, we could move this data into a dynamic struct that
> is only allocated on-demand. Potentially, that could decrease the size of
> most Items by 24 bytes, approx. 10% of the total struct size. Would you
> think the added complexity is worth it? Note that according to raw numbers,
> the total memory consumption would only go down by about one megabyte, less
> than 1% of the total peak consumption...

In ItemPrivate, mCachedPayloadParts, mVirtualReferences and maybe mTags are 
candidates for this optimization - they are rarely populated. Potentially some 
more saving could be done in EntityPrivate - mDeletedAttributes is rarely used 
too.  All this together might already have bigger impact on memory 
consumption, so it would be worth the little added code complexity.

> 
> # Question 2
> 
> Has anyone ever looked at Akonadi::ImapParser::parseQuotedString and
> ...::parseParenthesizedList, and is there an alternative? Afaik Volker told
> me that it would be gone with the streams API in Akonadi or something like
> that? It's amazing how often these functions are called - they triggers
> hundreds of thousands of memory allocations which can lead to memory
> fragmentation and generally slow down the performance.

I think Mark tried to look into it. I have ported the client libs to use 
ImapStreamParser (which currently lives in Akonadi server) - there are 
branches for that in Akonadi and kdepimlibs, but there are still some bugs, so 
they were not merged - now I'm waiting for the kdepimlibs split and Akonadi 
merge, so I can shuffle things around and start sharing more code between 
client libs and server.

> # Question 3
> 
> The memory consumption of KMime could probably be decreased by implicitly
> sharing the local part and domain in KMime::HeaderParsing::parseAddrSpec.
> Worth it?

Afaik this happens internally in KMime, so expiring the cache might be rather 
tricky. But I don't see that much into KMime, maybe it's possible.

> 
> # Question 4
> 
> Most items get a flag set of one item, containing the \SEEN flag, which
> wastes a lot of space. This could be implicitly shared (currently, the flag
> is shared, but not the QSet). Would it be OK to special-case the \SEEN flag
> set?

Sounds reasonable - if I measured it correctly, then QSet<QByteArray> with 
"\SEEN" is 136 bytes, so with ~200,000 read emails, we would save about 25 MB.

> 
> # Question 5
> 
> KMime::decodeRFC2047String is apparently decoding mailing list thread
> titles, among others. Again - this is a prime candidate for implicit
> sharing. The question again though is how to limit the cache lifetime.
> Afaik currently we leak the cache for mail adresses and domains and such.
> Could we maybe create per-collection caches and bind their lifetime to the
> time a collection's items are loaded in KMail?

Mark Gaiser was playing with this and we were planning to sit down on Akademy 
to figure out how to optimize the SerializerPluginMail. We were indeed 
struggling to solve the problem with cache expiration. Binding it's lifetime 
to collection lifetime sounds like a pretty good idea to me.


Cheers,
Daniel

> 
> # Wrap Up
> 
> I'd welcome feedback. If you want to reproduce this stuff, go download the
> sources for heaptrack and install it. It should be as simple as:
> 
> mkdir build;
> cd build;
> cmake -DCMAKE_BUILD_TYPE=Release ..;
> make install;
> heaptrack kmail
> heaptrack_print heaptrack.kmail....gz | less
> 
> And yes, I'll create a proper massif-visualizer integration *eventually* ;-)
> If this stuff is really really slow and perf top shows a ton of time spent
> in __GI___dl_iterate_phdr, patch libunwind with the patch at [3] and try
> again.
> 
> Cheers
> 
> [1]: http://quickgit.kde.org/?p=scratch%2Fmwolff%2Fheaptrack.git
> [2]: https://userpage.physik.fu-berlin.de/~milianw/kmail.mem.log
> [3]:
> https://userpage.physik.fu-berlin.de/~milianw/libunwind.better_performance.
> patch

-- 
Daniel Vrátil | dvratil at redhat.com | dvratil on #kde-devel, #kontact, #akonadi
KDE Desktop Team
Associate Software Engineer, Red Hat

GPG Key: 0xC59D614F6F4AE348
Fingerprint: 4EC1 86E3 C54E 0B39 5FDD B5FB C59D 614F 6F4A E348
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20140826/2a87e752/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/