[kdepim-users] rant

[Martin Steigerwald]

Am Mittwoch, 28. Januar 2015, 07:53:12 schrieb Pablo Sanchez:
> Hi,

Hi Pablo,

> As a Database and Performance engineer, I've investigated akonadi.
> 
> What I found at the database level is an excessive number of queries
> (IIRC, 1000's per second) which because all the data was in the DBMS'
> cache, manifested itself in 100% CPU utilization.

Did you test this with Akonadi git
c733429f4fa9696fb027ddc946e54f6bbb68deaf (branch 1.13)?

It contains gems like:

Avoid ridiculous amount of SQL queries by caching PartTypes

Implement cache for CollectionStatistics to significantly reduce amount of 
SQL queries

Avoid repeated calls to PimItem::flags() and PimItem::tags()

Avoid recursive collection listing in SearchHelper
 
> I used a VM on a 8p Host O/S.  The more processors I gave the VM, the
> more queries per second executed.  Which, on the face of it, one might
> say, great, our application is scaling well.  However, whenever I
> engage in this type of analysis, the question I always ask is what's
> the reason for the query which is our top consumer.
> 
> In this particular case, what I found was one of the top consuming
> queries was checking for new email.  1000's of times per second?  This
> makes no sense.  Even with a super high-speed connection, who cares!
> 
> :)
> 
> Daniel (sustaining engineer?) rolled in some changes to mitigate the
> incessant queries.  I don't know which version of Akonadi has these
> changes.

Yes, see above :)

And: It *does* help. I use it.

Still I think Akonadi caches too much and still has a lot of other issues. 
I posted quite some bug report numbers here already.

> IMO, the concept of Akonadi is a good one but the implementation
> (1000's of queries per second?) needs refactoring.  In the spirit of
> OSS, there will be /Akonadi Next/ .... with no disrespect intended,
> I'd prefer to see Akonadi fixed than to create a new set of potential
> issues.

I agree that the concept behind Akonadi is basically a good one.

But I also see the ideas for Akonadi Next as valid. While I know that 
using MySQL for handling the metadata of a ton of mails is possible as a 
groupware server like Zimbra shows by being able to access a folder with 
>480000 mails (thats right!) in the Web gui withing 5-10 seconds, I think 
that Akonadi has implementation issues. Sure, Zimbra tricks it, by just 
showing the first several thousand mails in the beginning and loaded more 
when you scroll down, but also the Lucense search index works 
marvellously. So, it *is* possible. This was with a VM with 12 GiB of RAM, 
but not just for one user, but for 15-50 users at once.

I think its possible to fix quite some of these implementation issues, but 
I think its multi process design leaves much room for synchronisation 
issues and may be difficult to get right. Also the scheduler within Akonadi 
is quite broke IMHO in blocking interactive user requests for background 
tasks, leaving KMail blocked for minutes. And it caches too much, way too 
much. 75 GiB in file_db_data is just too much IMHO.

I think Akonadi is in a similar situation as Nepomuk was after Vishesh´s 
optimization work. It often basically worked, but Baloo shows at what 
performance cost. Vishesh got to the point that he thought not being able 
to optimize it much more without replacing it.

So I think it may make sense to *simplify* it big time, while keeping the 
basic advantages. Whether it then uses MySQL or not, well… – while I do 
not like starting a MySQL for each user all that much – I don´t care that 
much on the other hand, if it works. But I do think the ideas of Akonadi 
Next make sense. And it may even make sense to use something else than a 
SQL database.

As does making Akonadi more robust and speedy, cause Akonadi Next will 
likely take some time.

Whatever gives:

- robustness (! very important, Akonadi lacks here, big time)
- speed
- simplicity and introspectability in case of issues

while keeping the advantages of Akonadi was to provide (but didn´t fully 
so far), like background processing *without* blocking apps, is fine with 
me.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM users mailing list
Subscription management: https://mail.kde.org/mailman/listinfo/kdepim-users