[Kde-pim] PIM is not good. Would adding an extra RC help?

Sat Apr 5 16:58:16 UTC 2014

Hi Albert,

I am mostly a user of KDEPIM, but since I experienced quite some performance 
issues and did some analysis and reports every now and then, I thought I share 
my findings.

Am Samstag, 5. April 2014, 16:40:24 schrieb Albert Astals Cid:
> ** Please CC me and r-t list, i'm not subscribed to pim list. **
> 
> Hello guys, as I hope you all know this Wednesday we are supposed to tag the
> final release for 4.13.
> 
> I am sending this email to ask for opinions on the value of adding an extra
> RC and delaying the 4.13.0 release for at least a week.
> 
> I'm going to explain why I'd like this to happen.
> 
> Since the update to 4.13 Beta I've been having lots of CPU and I/O problems
> in pim related stuff, most notably akonadi_baloo_indexer,
> akonadi_maildir_resource & friends.
> 
> On every release i've complained on IRC to be told "oh yes, we just fixed
> this thing and will be available on the next release".
> 
> But here I am, using 4.13 RC and with even KMail closed
> mysqld+akonadiserver+akonadi_maildir_resource are using 100% of one of my
> CPU cores and writing/reading around 10MB/s to my slow disks.

So you are using a POP3 setup? Or how is the maildir resource filled on your 
setup? How big is it?

I reported several bugs regarding slowness of maildir resource with lots of 
folders with lots of mails in it, including a report of 
akonadi_maildir_resource hogging up one Intel Sandybridge core on a Dual SSD 
BTRFS RAID 1 setup for several minutes after filtering about 100 mails into 
their folders[1]. And one for MySQL tuning, but at least from my observation 
the issue isn´t MySQL here, I thought so first, but I am not convinced after a 
deeper look at show innodb engine in MySQL is doing. See recent thread in pim-
ml as well[3]. I am keen to look at it some more, but after spending quite a 
lot of time on analysing I needed a break. Well some other things Mark 
found[4]

But on the other hand my setup might be considered extreme. About 100 folders, 
quite some of them with several ten thousand mails and one (kernel-ml) with 
about 220000 unread mails.

It is my impression that both for my private POP3 setup and my work IMAP setup 
performance got way better on the upgrade from Akonadi 1.11.0 to Akonadi 
1.12.0. Still I see the akonadi_maildir_resource hogging a core for minutes at 
times. But it doesn´t seem to happen after every mail retrieval / mail 
filtering cycle. Sometimes MySQL chimes in using something between one and 
three cores, but I see akonadi_maildir_resource for minutes on 100% CPU 
without any notable CPU activity of mysqld as well.

But when it does, I see it synchronizing folders. In strace I see it getting 
dentries and checking the existance of every little file.

I don´t know whether another week will lead to a fix. I got the impression that 
performance issues with Akonadi are not yet fully understood.

In my eyes there are two things which contribute a lot to KMail being 
basically unusable for times of several minutes:

1) Maildir resource synchronizes folders needlessly. Not sure what triggers it 
tough.

2) Akonadi blocks out KMail almost completely when maildir resource is doing 
so. To me Akonadi appears to be basically single threaded. While I understand 
one of its design goals of Akonadi were that GUI applications like KMail never 
block again.

Also remember for me this is Akonadi 1.12.0 and KDEPIM 4.12.4 still. But for 
these versions, although I see improvements, I still think KDEPIM + Akonadi 
needs serious performance related work. Cause I just don´t get how filtering 
100-200 mails to their folders can lead to a usage of one Sandybridge core for 
minutes unless I assume somewhere something does unnecessary work. Its not a 
I/O bottleneck according to atop.

That said:

- I don´t think the performance issues are new. So they are no regression. Or 
did you found it regressing recently?

- I am not sure whether another week would help with fixing them, as… my bet 
is: If the cause is understood someone likely would have fixed them already.

I do think the KDEPIM / Akonadi team needs helping hands. I tried to help a 
bit, but I didn´t come around at looking at the code and when I do I first need 
to understand some of it before I can consider proposing a patch. :)

[1] [Akonadi] [Bug 332684] New: [Maildir] lots of stats calls to 
/etc/localtime on synchronizing folders

=> can be work-arounded by setting TZ env var, yet folder synchronization 
still is slow.

[2] [Akonadi] [Bug 332626] MySQL tuning: adaption of MySQL tuning options for 
larger accounts

[3] [Kde-pim] Akonadi MySQL backend: tuning for larger accounts or switching 
to MariaDB with a different storage engine?

[4] PIM Sprint, Calendar progress and Akonadi
Datum:Donnerstag 17:13 
Autor:Mark Gaiser (markg)

http://kdeblog.mageprojects.com/2014/04/03/pim-sprint-calendar-progress-and-akonadi/

[5] Re: [Kde-pim] Akonadi MySQL backend: tuning for larger accounts or 
switching to MariaDB with a…

This is not the beginning of the thread, but lists.kde.org is very slow at the 
moment.

http://lists.kde.org/?l=kde-pim&m=139603704423385&w=2

[Kde-pim] Slowness in POP3 case, maildir resource stat()ing on /etc/localtime 
very often…
http://lists.kde.org/?l=kde-pim&m=139592803810131&w=2

[6] [Kde-pim] Also fetches all items on Maildir filtering? (was: Review Request 
117140: ItemSync: Only…
http://lists.kde.org/?l=kde-pim&m=139629170212642&w=2

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7