Review Request 117975: Change libmaildir listNew and listCurrent functions to explicetely not sort QDir.entryList

[Martin Steigerwald]

> On May 29, 2014, 12:33 p.m., Commit Hook wrote:
> > This review has been submitted with commit ea0b761fbe3c43b42f83c5477018e10209f92ea3 by Martin Steigerwald to branch KDE/4.13.
> 
> Martin Steigerwald wrote:
>     My changes breaks two unittests:
>     
>              16 - maildir-testmaildir (Failed)
>              30 - mixedmaildir-itemfetchtest (Failed)
>     
>     I researched the maildir related one:
>     
>     FAIL!  : MaildirTest::testMaildirFlagsReading() Compared values are not the same
>        Actual (flags.count()): 0
>        Expected (1): 1
>        Loc: [/home/martin/KDE/Dev/kdepim-runtime/resources/maildir/libmaildir/tests/testmaildir.cpp(363)]
>     
>     362   Akonadi::Item::Flags flags = d.readEntryFlags( entries[0] );
>     363   QCOMPARE( flags.count(), 1 );                                                 
>     364   QVERIFY( flags.contains( Akonadi::MessageFlags::Forwarded ) );
>     
>     I don´t get why it fails. I thought due to keycache.cpp requiring sorting in methods listNew and listCurrent, but readEntryFlags doesn´t seem to call into keycache. Any hints?
>     
>     I will revert patches at least from KDE/4.13 branch if I don´t find why its breaking and thus not able to fix it.
>     
>     Heck, I think I got it:
>     
>     The test uses: entries[0] which isn´t guarenteed to test the right entry in an unsorted list.
> 
> Martin Steigerwald wrote:
>     Review request 118400 for libmaildir test case.

Review request #118441 for mixedmaildir test case.


- Martin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/117975/#review58710
-----------------------------------------------------------


On May 20, 2014, 10:40 p.m., Martin Steigerwald wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/117975/
> -----------------------------------------------------------
> 
> (Updated May 20, 2014, 10:40 p.m.)
> 
> 
> Review request for KDEPIM, KDEPIM-Libraries, Andras Mantia, David Faure, and Sergio Luis Martins.
> 
> 
> Bugs: 334218
>     http://bugs.kde.org/show_bug.cgi?id=334218
> 
> 
> Repository: kdepim-runtime
> 
> 
> Description
> -------
> 
> On trying to find the cause for Akonadi maildir resource hogging up a Sandybridge core for minutes without any notable MySQL or disk activity, Sergio recommended doing a callgrind run. This revealed a bottleneck with QDir.entryList and there specifically in QAlgorihmsPrivate::qSortHelper[1].
> 
> Calling setSorting(QDir::NoSort) on the QDir object gets rid of the sorting. This I did twice in maildir.ccp and twice in keycache.cpp for listCurrent and listNew functions there.
> 
> It is not yet clear whether this change is safe, but on my tests it appears to do. Moving and filtering mails still work okay. I will CC Andras on this as Sergio tried to ping him about this change.
> 
> I think it is fair to say that this change has a *huge* impact on Akonadi maildir performance with large maildir and I recommend to backport this for stable branch as well once changing to no sorting is considered to be safe.
> 
> Thanks to Sergio and David for pointing out the sorting issue, telling how to change to no sorting and general help.
> 
> See also:
> 
> Bug 334218 - synchronizations of large folders with filesystem contents hogs a Sandybridge core for minutes
> https://bugs.kde.org/334218
> 
> [1] https://bugs.kde.org/show_bug.cgi?id=334218#c4
> 
> [2] Bug 334206 - While maildir resources synchronizes a folder KMail blocks on switching to a different folder:
> https://bugs.kde.org/334206
> 
> 
> Diffs
> -----
> 
>   resources/maildir/libmaildir/keycache.cpp f0af9c47cef63dac45ae68dd94fa17a040dc1593 
>   resources/maildir/libmaildir/maildir.cpp 9bd380201a1a250ecfe354dd99946e8dae0ab668 
> 
> Diff: https://git.reviewboard.kde.org/r/117975/diff/
> 
> 
> Testing
> -------
> 
> In my tests with KMail SC 4.12.4 Debian unstable packages and Akonadi and kdepimlibs from Git it works well. Well basically Akonadi maildir resource changed from being a CPU hog for minutes to hardly every appearing in a 10 second measurement average snapshot of atop anymore. KMail hardly blocks on folder changes anymore, which I still believe to be a different issue tough[2].
> 
> Mail filtering and moving mails to a different folder still works ago. And KMail is subjectively much faster. It completely changes my KMail experience from being unbearable at times to quite pleasant.
> 
> 
> Thanks,
> 
> Martin Steigerwald
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20140531/76e13d14/attachment.html>