"Retrieving Folders..." hang and a possible solution

[ianseeks]

On Sunday, 5 February 2017 11:27:06 GMT Martin Steigerwald wrote:
> Adding in kde-pim, so KDEPIM devs have a chance to notice this and give
> feedback if they wish to.
> 
> Am Sonntag, 5. Februar 2017, 09:30:21 CET schrieb ianseeks:
> > Hi
> > 
> > I've posted this to 3 lists in case it helps someone and stop blaming
> > kmail, maybe there is a case for akonadi to identify a resource issue or
> > a decent default should be in the base install.
> 
> Stop blaming KMail for delays is often a good idea.
> 
> There is mainly two kinds of delays that are actually caused by KMail:
> 
> 1) Display (not retrieval!) of the contents of large folders, especially
> when threading view mode is enabled. Dan, but I also think other PIM
> developers *heavily* optimized this. KMail is much, much, much here faster
> since about KDEPIM 16.04.
> 
> 2) Selecting all mails in one folder in order to move tham or do something
> else with them. It already takes several seconds of KMail hogging one CPU
> core when selecting all mails of a folder with about 15000 mails on this
> Sandybridge ThinkPad here. I think when dragging those mails to a different
> folder there is another delay. There are further delays then caused by
> Akonadi which are much higher. This is way I do moving folders with lots of
> mails differently meanwhile: Stop Akonadi, mv the folder, start Akonadi and
> let it update the database. This is still not optimal but lots faster than
> during it through KMail. (I also reported this one.)
> 
> To my experience all or almost all other delays are caused by either Akonadi
> or in case of IMAP accounts slow access to IMAP server, which may partly
> also be caused by Akonadi hammering the IMAP server during (probably
> needless) synchronisation runs.
> 
> > I originally reduced the size of my mail folders by deleting everything
> > prior to 2016 then deleted the akonadi database in the hope of fixing the
> > problem but no go. So i created a new user with a brand new mail database
> > and still the problem persisted.
> > 
> > Martin Steigerwald suggested i try this and it seems to be working for me,
> > i've not had the hanging "Retrieving Folders..." since i did it yesterday
> > afternoon.
> > 
> > I stopped akonadiserver and mysqld (if it doesn't do it by itself).
> > I increased the innodb_buffer_pool_size value from 80M to 128M in
> > ~/.local/
> > share/akonadi/mysql.conf. (it was set to 80M which is 48 below the default
> > - not sure who set below the mysql default).  This new value may not be
> > enough for your circumstances.
> > 
> > Start akonadi again.
> > 
> > Akonadi still crashes from time to time, especially on log out, but i
> > think
> > its for other reasons now.
> 
> I brought this up quite a time ago already, but didn´t find any easily
> measurable difference back then:
> 
> [Akonadi] [Bug 332626] MySQL tuning: adaption of MySQL tuning options for
> larger accounts
> https://bugs.kde.org/show_bug.cgi?id=332626

You wrote a fairly comprehensive explanation on that bug but there is not one 
comment from a dev to even say "i just looked at it" which makes you think 
that its been ignored.  i wish kde.bugs reported somewhere on the bug how many 
have looked at it and who has actually opened it if they are in the "send to" 
list, it would be more assuring.
 
> I still didn´t measure, but meanwhile I have the firm impression that it
> really helps with larger accounts to raise that size. Even to 1 GiB.
> 
> So my recommendation would give it a good increase. I never systematically
> tested different sizes… it may well be that for my usecase 512 MiB would or
> even be enough.
> 
> My oppinion still is: It is a bug that an user would have to care for
> approbiate database tuning. Akonadi serves desktop applications like KMail
> and its unreasonable for to expect that every KMail user also is a database
> administrator.

Totally agree there.  I would never have got your buffer pool solution by 
myself.

> Just raising the setup by default does not serve those systems with little
> amounts of RAM. I´d personally for the time going would go with a moderate
> increase of the default, since from all I know about database performance,
> 80 MiB of InnoDB buffer pool size is ridiculously low.

I would have thought it should have at least been left as the mysql default 
value rather than a lower value.

> Also this is and quite other bug reports are mentioning Retrieving folder
> contents issue:
> 
> https://bugs.kde.org/show_bug.cgi?id=338571
> 
> However AFAIK there is a combination of several issues at place:
> 
> 1) If Akonadi does a folder synchronisation, it doesn´t do anything else,
> i.e. blocks out mail access:
> 
> [Akonadi] [Bug 367892] New: During folder synchronisation Akonadi blocks out
> other operations like deleting or viewing mails
> 
> [kmail2] [Bug 331848] displaying, moving, deleting mails takes 10-20 seconds
> when Akonadi synchronizes in background
> 
> Hmmm, I think these are duplicates. Will look and mark so.
> 
> I think this is a major issue: A background operation is not supposed to
> block out interactive user requests *ever*. I think you know how
> frustrating it is to wait for Akonadi to talk with KMail again.
> 
> 2) IMHO it also needlessly does folder synchronisation:
> 
> [Akonadi] [Bug 334216] New: synchronizes folder with filesystem after
> downloading and filtering mails needlessly
> 
> (note, that exactly with hogging one core for minutes usually is no more due
> to other optimization in database access, and its folder synchronisation
> causes less I/O and thus is faster due to high innodb buffer size I think)
> 
> This is the maildir case. I think I also reported it for IMAP. I just do not
> find this bug report anymore, but it was very easy to reproduce: Have a
> fast IMAP server like dovecot. Have a folder where you can safely delete
> from mails from. Delete mails one by one using delete key. Watch KMail
> synchronizing the folder after about 5-10 delete operations, consequently
> blocking out other operations.
> 
> 3) Too many and too complex database accesses.
> 
> Dan did some great work there, so by all means use at least KDEPIM + Akonadi
> 16.04. But its still doing queries it shouldn´t do according to what I
> read.
> 
> 4) Non optimal database configuration for large accounts.
> 
> 
> Of all of these issues the user and to some extent distros can directly only
> influence 4. I.e. mainly raise InnoDB buffer pool size. So this is what I
> recommend users to try. However: It cannot and will not solve all
> performance issues. But in my experience it can give a much better
> experience.
> 
> Unfortunately performance in Akonadi is a quite complex topic, which
> requires further serious work on it. Already for just diagnosing the actual
> bottle necks. Database admins offered their help before and maybe Randa
> Meetings would be a good place to meet up regarding this. I consider
> attending and bring with me my insane Akonadi testcase (and in my case: its
> really insane, with a working set of more than 940000 mails and at least
> another 1500000 mails I somehow tricked Akonadi into not synchronizing
> unless I click on these archival folders.)
> 
> Thanks,


-- 
opensuse:tumbleweed:20170203
Qt: 5.7.1
KDE Frameworks: 5.30.0
KDE Plasma: 5.9.0
kwin5-5.9.0-1.1.x86_64
kmail2 5.4.1
Kernel:  4.9.6-1-default
Nouveau:  1.0.13_2.2