Review Request 117975: Change libmaildir listNew and listCurrent functions to explicetely not sort QDir.entryList

[Martin Steigerwald]

> On May 5, 2014, 3:08 p.m., Sergio Luis Martins wrote:
> > Commit to stable too ?
> 
> Laurent Montel wrote:
>     Please wait after 4.13.1 it will tag the 8 may.
>     Apply it in 4.13.2 if safe
>     thanks
> 
> Martin Steigerwald wrote:
>     Okay, will wait. I think its good if someone else than me actually also tests it. For me it works really nicely.

Hi! So did anyone test with it? It works very well for me. I see higher I/O throughput on accessing folders, but I think thats just due to the fact that there is way less CPU overhead now. It will stat() mail files in random order instead of alphabetically now, I think. Depending on the filesystem implementation and the order in which those files will be created, this may make a difference – accessing in inode number order may help on some filesystems. In any case, subjectively I think it is way better than before. What do you experience?

So did anyone test and okay with committing to 4.13.2? How can I do it? Rebase my nosorting branch to 4.13.2 branch and then merge my nosorting branch with the two commits into 4.13.2 branch? I did so with my commits to master branch, so I think this would work.


- Martin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/117975/#review57321
-----------------------------------------------------------


On May 5, 2014, 12:07 p.m., Martin Steigerwald wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/117975/
> -----------------------------------------------------------
> 
> (Updated May 5, 2014, 12:07 p.m.)
> 
> 
> Review request for KDEPIM, KDEPIM-Libraries, Andras Mantia, David Faure, and Sergio Luis Martins.
> 
> 
> Bugs: 334218
>     http://bugs.kde.org/show_bug.cgi?id=334218
> 
> 
> Repository: kdepim-runtime
> 
> 
> Description
> -------
> 
> On trying to find the cause for Akonadi maildir resource hogging up a Sandybridge core for minutes without any notable MySQL or disk activity, Sergio recommended doing a callgrind run. This revealed a bottleneck with QDir.entryList and there specifically in QAlgorihmsPrivate::qSortHelper[1].
> 
> Calling setSorting(QDir::NoSort) on the QDir object gets rid of the sorting. This I did twice in maildir.ccp and twice in keycache.cpp for listCurrent and listNew functions there.
> 
> It is not yet clear whether this change is safe, but on my tests it appears to do. Moving and filtering mails still work okay. I will CC Andras on this as Sergio tried to ping him about this change.
> 
> I think it is fair to say that this change has a *huge* impact on Akonadi maildir performance with large maildir and I recommend to backport this for stable branch as well once changing to no sorting is considered to be safe.
> 
> Thanks to Sergio and David for pointing out the sorting issue, telling how to change to no sorting and general help.
> 
> See also:
> 
> Bug 334218 - synchronizations of large folders with filesystem contents hogs a Sandybridge core for minutes
> https://bugs.kde.org/334218
> 
> [1] https://bugs.kde.org/show_bug.cgi?id=334218#c4
> 
> [2] Bug 334206 - While maildir resources synchronizes a folder KMail blocks on switching to a different folder:
> https://bugs.kde.org/334206
> 
> 
> Diffs
> -----
> 
>   resources/maildir/libmaildir/keycache.cpp f0af9c47cef63dac45ae68dd94fa17a040dc1593 
>   resources/maildir/libmaildir/maildir.cpp 9bd380201a1a250ecfe354dd99946e8dae0ab668 
> 
> Diff: https://git.reviewboard.kde.org/r/117975/diff/
> 
> 
> Testing
> -------
> 
> In my tests with KMail SC 4.12.4 Debian unstable packages and Akonadi and kdepimlibs from Git it works well. Well basically Akonadi maildir resource changed from being a CPU hog for minutes to hardly every appearing in a 10 second measurement average snapshot of atop anymore. KMail hardly blocks on folder changes anymore, which I still believe to be a different issue tough[2].
> 
> Mail filtering and moving mails to a different folder still works ago. And KMail is subjectively much faster. It completely changes my KMail experience from being unbearable at times to quite pleasant.
> 
> 
> Thanks,
> 
> Martin Steigerwald
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20140511/f247a0e9/attachment.html>