Goodbye for now, kmail

Sat May 6 09:28:59 BST 2017

Daniel Vrátil - 06.05.17, 01:09:
> On Friday, May 5, 2017 4:45:08 PM CEST Martin Steigerwald wrote:
> > Pablo Sanchez - 05.05.17, 09:06:
> > > [ Comments below, in-line ]
> > > 
> > > On Fri, 05 May 2017 17:57:39 +0530, Vishnu V. Krishnan wrote:
[…]
> > - Akonadi does needless work – like to frequent folder and too expensive
> > synchronisations (on IMAP after every few mails you delete). It requests
> > all DB entries for the collection it synchronizes to compare what is
> > stored in backend storage like IMAP or local Maildir.
> 
> This is only true for Maildir and mixedmaildir resources. There are
> certainly other more efficient ways how to do this nowadays, someone just
> needs to look into it. As always, patches are welcomed.

Well I can only speak of KDEPIM 16.04 and Akonadi 16.04, but last time I tried 
with that version, a folder synchronisation was triggered *every single time* 
after pressing Delete key one about 5-10 mails. I seriously do not get why 
this should be necessary. I do not have any deep knowledge about the IMAP 
protocol, but I´d certainly expect that after a delete operation completed it 
will report back whether it was successful, and when it was successful there 
is no need to trigger a folder synchronizartions.

> > For local maildir I would like
> > to disable automatic folder synchronization *completely* cause I know that
> > Akonadi Maildir is the only user of it, yet it checks frequently whether
> > the million of mail files is still there.
> 
> Just turn off Automatic Synchronization for the folder. If the sync is still
> triggered, then it's triggered due to some external factor, and it needs to
> be debugged what triggers it.

I needed to dig a bit for this setting, but I see, I did so in the past 
already:

It is turned off for maildir resource main folder… and all subfolders I 
checked inherit that setting. I do have the following setting:

- Synchronize upon selecting the folder (roughly translated, running KMail in 
german) is checked

- Automatic synchronisation is set to never

Then I also set it to only retrieve the content of a message when it is 
necessary.

So that would be a new issue, if not already reported, I think.

> > - Akonadi does more needless work: When you moving a folder from one
> > parent
> > folder to another parent folder within a local Maildir resource it first
> > creates the new folder, then copies all mails, then deletes it from the
> > source folder, essentially moving them and then deletes the source folder…
> > instead of just *renaming* the folder.
> 
> Not true. The folder is renamed (see MaildirResource::collectionMoved() and
> MailDir::moveTo()).

Definately true for KDEPIM and Akonadi 16.04. I just tested it again. Akonadi 
hosts a CPU and I/O party on doing that with a folder of 50000 mails and I can 
only type really slow now while it does so. Now mysqld at 100% of one core, 
kmail at 84% of one core for more than 1 minute already. Okay KMaal has been 
involved, cause I had source folder still open. Additionally it appears to 
trigger Akonadi Indexing Agent again for the mails.

I know this so well, cause as a workaround, I looked up the folder in the 
filesystem and moved it there. This still triggers quite a database workload, 
as Akonadi detects it through inotify watches.

So has this been fixed meanwhile? I hope to upgrade to a later version soon.

After having written all this its still not finished and mysqld still uses up 
100% of a folder. This time I didn´t look specifically what Akonadi is doing, I 
did this for my bug report, needing several minutes of CPU usage for  just 
renaming a folder? I don´t believe this.

> > - Akonadi does not auto-tune MySQL performance settings like InnoDB buffer
> > pool size with is ridiculously low after anything than a few mails. A too
> > low InnoDB buffer pool size leads to additional file I/O accesses.
> 
> innodb_buffer_pool_size has been doubled in 17.04 release (to 128MB).

Wasn´t it 80 MiB before? Anyway, 128 MiB is still much too low for larger 
setups. And unless you want users to be a database admin, I think Akonadi 
should autotune it.

> > Thing is: All of these issues are known since years. And while Dan, Milan
> > and others did impressive performance optimizations these basic issues are
> > all still unfixed.
> 
> Patches are welcomed.

I understand. Honestly I don´t see myself developing patches at the moment. 
Due to time constraint, but also due to the fact that the last time I tried to 
understand Akonadi code structure and understand enough of how it works… I 
didn´t really get it.

Also I am not fully convinced that Akonadi is the right design to approach the 
challenges it is aimed to address. But what can I say… from what I found so 
far in its current incarnation I can *prove* it to do needless work in so many 
areas that it is not really funny. At least with 16.04. So I don´t really buy 
enough into Akonadi to be convinced I can do a meaningful contribution to it 
easily enough again. I did once, but I received a lot of help to profile the 
code back then… as I stopped maildir resource from *sorting* (!) all filenames 
of the maildir directory it synchronizes for *no* apparent reason. That was my 
one and up to now only commit to Akonadi. I wonder how many of those nuggets 
that can give *huge* performance improvements are still in the code.

However I intend to upgrade to newer KDEPIM soon. Hopefully with newer Debian 
packages after release of Stretch… but maybe even by using kdesrc-build again.

Also I´d need to learn more about C++ in order to contribute efficiently and 
meaningfully enough. I had the idea to do so and I do have some holidays, so 
maybe I will look for a good online resource to learn it.

> > And as such, while KMail and Akonadi works well enough for me after
> > raising
> > InnoDB buffer pool size to 1 GiB, and believe me I can easily notice when
> > on an Akonadi upgrade it gets downgraded to 80 MiB again…, I can fully
> > understand the massive frustration with it. Usually I have retrieving
> > folder contents or mail notifications only for a few seconeds, but I am
> > still seeing this. And this is on Dual SSD BTRFS RAID 1 with surely fast
> > enough Sandybridge ThinkPad.
> > 
> > 
> > So while I believe that creating a solution that works and uses an DBMS is
> > possible… for some reason be it lack of developer time and/or design
> > issues
> > within Akonadi, Akonadi still is not that solution. And thats after more
> > than 5 years of introducing it. So I perfectly can understand the
> > frustration and I certainly also can understand the arguments to ditch
> > Akonadi completely or replace it by something else like Sink. However Sink
> > probably did not prove itself in the scenarios KDEPIM would need it for.
> 
> Sink has some very good concepts that I like and we will be gradually
> evolving Akonadi towards those concepts. But I don't want to walk down the
> route of "let's throw everything away and start from scratch" though.

Well I bet it would be a lot of work to adapt all the KDEPIM applications that 
use Akonadi to use Sink instead. I did not look into Sink so far… and I don´t 
fully understand Akonadi´s working, so while I am reluctant about Akonadi just 
some seeing that years after its introduction it still does not deliver for 
many users, I have no idea what would be the best way to move forward and am 
sure you have a better understanding of the benefits and shortcomings of each 
approach.

I close with the remark that this 50000 mail folder *still* is not moved and 
mysqld still uses 100% of CPU. On moving back that folder I bet I will use my 
old move it in the filesystem trick again. At least that operation appears to 
be running in the background now and KMail is responsive enough.

Thanks,
-- 
Martin