[Kde-pim] Optimizing KMail's memory usage

[Mark Gaiser]

On Tue, Aug 26, 2014 at 11:55 AM, Daniel Vratil <dvratil at redhat.com> wrote:
> On Monday 25 of August 2014 20:16:25 Milian Wolff wrote:
>> Hello all,
>
> Hi Millian,
>
> first - thanks a lot for looking into this and suggesting improvements - we
> need this a lot :-)
>
>>
>> I'd like to propose a change to decrease the memory consumption of KMail.
>> Akonadi::ItemPrivate in kdepimlibs/akonadi/item_p.h is currently the number
>> one memory consumer. As evidenced by the heaptrack [1] log I generated, see
>> [2], 9191232 bytes peak-memory consumption come from Akonadi::ItemPrivate
>> (search for "PEAK MEMORY CONSUMERS" in the output log). The class is pretty
>> large, i.e. 216bytes on my 64 bit machine, and is created extremely often
>> (if you have enough mails, that is ;-)).
>>
>> # Question 1
>>
>> Looking at its data members, I have an idea how to decrease the size: It
>> seems as if it reserves a lot of space for *potential* changes, such as:
>>
>> Item::Flags mAddedFlags;
>> Item::Flags mDeletedFlags;
>> Tag::List mAddedTags;
>> Tag::List mDeletedTags;
>>
>> I wonder - how often is an Item actually changed? If it is rarely, as I'd
>> think should be the case, we could move this data into a dynamic struct that
>> is only allocated on-demand. Potentially, that could decrease the size of
>> most Items by 24 bytes, approx. 10% of the total struct size. Would you
>> think the added complexity is worth it? Note that according to raw numbers,
>> the total memory consumption would only go down by about one megabyte, less
>> than 1% of the total peak consumption...
>
> In ItemPrivate, mCachedPayloadParts, mVirtualReferences and maybe mTags are
> candidates for this optimization - they are rarely populated. Potentially some
> more saving could be done in EntityPrivate - mDeletedAttributes is rarely used
> too.  All this together might already have bigger impact on memory
> consumption, so it would be worth the little added code complexity.
>
>>
>> # Question 2
>>
>> Has anyone ever looked at Akonadi::ImapParser::parseQuotedString and
>> ...::parseParenthesizedList, and is there an alternative? Afaik Volker told
>> me that it would be gone with the streams API in Akonadi or something like
>> that? It's amazing how often these functions are called - they triggers
>> hundreds of thousands of memory allocations which can lead to memory
>> fragmentation and generally slow down the performance.
>
> I think Mark tried to look into it. I have ported the client libs to use
> ImapStreamParser (which currently lives in Akonadi server) - there are
> branches for that in Akonadi and kdepimlibs, but there are still some bugs, so
> they were not merged - now I'm waiting for the kdepimlibs split and Akonadi
> merge, so I can shuffle things around and start sharing more code between
> client libs and server.

yes, that was me :)

I want to take a look at that code path again once the stream stuff is
merged in. Speeding up this part also speeds up indexing.

>
>> # Question 3
>>
>> The memory consumption of KMime could probably be decreased by implicitly
>> sharing the local part and domain in KMime::HeaderParsing::parseAddrSpec.
>> Worth it?
>
> Afaik this happens internally in KMime, so expiring the cache might be rather
> tricky. But I don't see that much into KMime, maybe it's possible.
>
>>
>> # Question 4
>>
>> Most items get a flag set of one item, containing the \SEEN flag, which
>> wastes a lot of space. This could be implicitly shared (currently, the flag
>> is shared, but not the QSet). Would it be OK to special-case the \SEEN flag
>> set?
>
> Sounds reasonable - if I measured it correctly, then QSet<QByteArray> with
> "\SEEN" is 136 bytes, so with ~200,000 read emails, we would save about 25 MB.
>
>>
>> # Question 5
>>
>> KMime::decodeRFC2047String is apparently decoding mailing list thread
>> titles, among others. Again - this is a prime candidate for implicit
>> sharing. The question again though is how to limit the cache lifetime.
>> Afaik currently we leak the cache for mail adresses and domains and such.
>> Could we maybe create per-collection caches and bind their lifetime to the
>> time a collection's items are loaded in KMail?
>
> Mark Gaiser was playing with this and we were planning to sit down on Akademy
> to figure out how to optimize the SerializerPluginMail. We were indeed
> struggling to solve the problem with cache expiration. Binding it's lifetime
> to collection lifetime sounds like a pretty good idea to me.
>
>
> Cheers,
> Daniel
>
>>
>> # Wrap Up
>>
>> I'd welcome feedback. If you want to reproduce this stuff, go download the
>> sources for heaptrack and install it. It should be as simple as:
>>
>> mkdir build;
>> cd build;
>> cmake -DCMAKE_BUILD_TYPE=Release ..;
>> make install;
>> heaptrack kmail
>> heaptrack_print heaptrack.kmail....gz | less
>>
>> And yes, I'll create a proper massif-visualizer integration *eventually* ;-)
>> If this stuff is really really slow and perf top shows a ton of time spent
>> in __GI___dl_iterate_phdr, patch libunwind with the patch at [3] and try
>> again.
>>
>> Cheers
>>
>> [1]: http://quickgit.kde.org/?p=scratch%2Fmwolff%2Fheaptrack.git
>> [2]: https://userpage.physik.fu-berlin.de/~milianw/kmail.mem.log
>> [3]:
>> https://userpage.physik.fu-berlin.de/~milianw/libunwind.better_performance.
>> patch
>
> --
> Daniel Vrátil | dvratil at redhat.com | dvratil on #kde-devel, #kontact, #akonadi
> KDE Desktop Team
> Associate Software Engineer, Red Hat
>
> GPG Key: 0xC59D614F6F4AE348
> Fingerprint: 4EC1 86E3 C54E 0B39 5FDD B5FB C59D 614F 6F4A E348
> _______________________________________________
> KDE PIM mailing list kde-pim at kde.org
> https://mail.kde.org/mailman/listinfo/kde-pim
> KDE PIM home page at http://pim.kde.org/
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/