"Retrieving Folders..." hang and a possible solution

[Martin Steigerwald]

Adding in kde-pim, so KDEPIM devs have a chance to notice this and give 
feedback if they wish to.

Am Sonntag, 5. Februar 2017, 09:30:21 CET schrieb ianseeks:
> Hi
> 
> I've posted this to 3 lists in case it helps someone and stop blaming kmail,
> maybe there is a case for akonadi to identify a resource issue or a decent
> default should be in the base install.

Stop blaming KMail for delays is often a good idea.

There is mainly two kinds of delays that are actually caused by KMail:

1) Display (not retrieval!) of the contents of large folders, especially when 
threading view mode is enabled. Dan, but I also think other PIM developers 
*heavily* optimized this. KMail is much, much, much here faster since about 
KDEPIM 16.04.

2) Selecting all mails in one folder in order to move tham or do something 
else with them. It already takes several seconds of KMail hogging one CPU core 
when selecting all mails of a folder with about 15000 mails on this 
Sandybridge ThinkPad here. I think when dragging those mails to a different 
folder there is another delay. There are further delays then caused by Akonadi 
which are much higher. This is way I do moving folders with lots of mails 
differently meanwhile: Stop Akonadi, mv the folder, start Akonadi and let it 
update the database. This is still not optimal but lots faster than during it 
through KMail. (I also reported this one.)

To my experience all or almost all other delays are caused by either Akonadi 
or in case of IMAP accounts slow access to IMAP server, which may partly also 
be caused by Akonadi hammering the IMAP server during (probably needless) 
synchronisation runs.

> I originally reduced the size of my mail folders by deleting everything
> prior to 2016 then deleted the akonadi database in the hope of fixing the
> problem but no go. So i created a new user with a brand new mail database
> and still the problem persisted.
> 
> Martin Steigerwald suggested i try this and it seems to be working for me,
> i've not had the hanging "Retrieving Folders..." since i did it yesterday
> afternoon.
> 
> I stopped akonadiserver and mysqld (if it doesn't do it by itself).
> I increased the innodb_buffer_pool_size value from 80M to 128M in ~/.local/
> share/akonadi/mysql.conf. (it was set to 80M which is 48 below the default -
> not sure who set below the mysql default).  This new value may not be
> enough for your circumstances.
> 
> Start akonadi again.
> 
> Akonadi still crashes from time to time, especially on log out, but i think
> its for other reasons now.

I brought this up quite a time ago already, but didn´t find any easily 
measurable difference back then:

[Akonadi] [Bug 332626] MySQL tuning: adaption of MySQL tuning options for 
larger accounts
https://bugs.kde.org/show_bug.cgi?id=332626

I still didn´t measure, but meanwhile I have the firm impression that it really 
helps with larger accounts to raise that size. Even to 1 GiB.

So my recommendation would give it a good increase. I never systematically 
tested different sizes… it may well be that for my usecase 512 MiB would or 
even be enough.

My oppinion still is: It is a bug that an user would have to care for 
approbiate database tuning. Akonadi serves desktop applications like KMail and 
its unreasonable for to expect that every KMail user also is a database 
administrator.

Just raising the setup by default does not serve those systems with little 
amounts of RAM. I´d personally for the time going would go with a moderate 
increase of the default, since from all I know about database performance, 80 
MiB of InnoDB buffer pool size is ridiculously low.

Also this is and quite other bug reports are mentioning Retrieving folder 
contents issue:

https://bugs.kde.org/show_bug.cgi?id=338571

However AFAIK there is a combination of several issues at place:

1) If Akonadi does a folder synchronisation, it doesn´t do anything else, i.e. 
blocks out mail access:

[Akonadi] [Bug 367892] New: During folder synchronisation Akonadi blocks out 
other operations like deleting or viewing mails

[kmail2] [Bug 331848] displaying, moving, deleting mails takes 10-20 seconds  
when Akonadi synchronizes in background

Hmmm, I think these are duplicates. Will look and mark so.

I think this is a major issue: A background operation is not supposed to block 
out interactive user requests *ever*. I think you know how frustrating it is 
to wait for Akonadi to talk with KMail again.

2) IMHO it also needlessly does folder synchronisation:

[Akonadi] [Bug 334216] New: synchronizes folder with filesystem after 
downloading and filtering mails needlessly

(note, that exactly with hogging one core for minutes usually is no more due 
to other optimization in database access, and its folder synchronisation 
causes less I/O and thus is faster due to high innodb buffer size I think)

This is the maildir case. I think I also reported it for IMAP. I just do not 
find this bug report anymore, but it was very easy to reproduce: Have a fast 
IMAP server like dovecot. Have a folder where you can safely delete from mails 
from. Delete mails one by one using delete key. Watch KMail synchronizing the 
folder after about 5-10 delete operations, consequently blocking out other 
operations.

3) Too many and too complex database accesses.

Dan did some great work there, so by all means use at least KDEPIM + Akonadi 
16.04. But its still doing queries it shouldn´t do according to what I read.

4) Non optimal database configuration for large accounts.


Of all of these issues the user and to some extent distros can directly only 
influence 4. I.e. mainly raise InnoDB buffer pool size. So this is what I 
recommend users to try. However: It cannot and will not solve all performance 
issues. But in my experience it can give a much better experience.

Unfortunately performance in Akonadi is a quite complex topic, which requires 
further serious work on it. Already for just diagnosing the actual bottle 
necks. Database admins offered their help before and maybe Randa Meetings 
would be a good place to meet up regarding this. I consider attending and 
bring with me my insane Akonadi testcase (and in my case: its really insane, 
with a working set of more than 940000 mails and at least another 1500000 
mails I somehow tricked Akonadi into not synchronizing unless I click on these 
archival folders.)

Thanks,
-- 
Martin