[Kde-pim] Akonadi: single database design mistake?

[Martin Steigerwald]

Hi Dmitry!

I rarely post here as a user of KDEPIM, but this I like to comment:

Am Dienstag, 29. November 2011 schrieb Dmitry Torokhov:
> > Another thing: If we ever come to the point where MySQL is the
> > bottleneck, the  current architecture should make it rather simple
> > to come up with an alternative, optimized architecture. Personally,
> > I just doubt that we are able to design a relational database from
> > scratch that will outperform MySQL so easily...
> 
> Thet is the main question: do we really want a monolithic database
> here.

At work we use a Zimbra Collaboration Suite server as our groupware 
solution. It uses MySQL to store mail metadata, Lucene to provide a search 
index and files to store the actual mail. It is serving about 30 users all 
day via Zimbra webclient, outlook and various IMAP clients - I partly use 
KMail with it.

Now consider this:

- about 100 folders including subfolders
- hundred of thousands of mails
- several gigabytes of mail easily (do not see it in the webclient and do 
not want to open up a VPN to look in the administrative interface for the 
size right now)
- folders with tens of thousands of mails

And thats just my account there.

Now I have a nice little folder here:

- kernel-ml (270997)

Yes that in the brackets is count of unread mail. In *one* folder.

Now lets click on it to see the index. I am using the AJAX based 
webclient.

You may now guess how long it takes to see it.

Guess it first, then scroll down!










































































About 2-5 seconds!

Yes, read again: About 2-5 seconds!

Granted the webclient tricks about it. It only shows the most recent 
mails. Only when I scroll down it shows more mails. But then it has to 
know which mails the most recent are so it had to search the index of the 
folder for them.

Also granted that it might take upto 15 seconds when the server is on more 
load, when more co-workers access it as well that at this time. But then I 
won´t have this on my laptop with Akonadi.

I use KMail 1 from KDE 4.6.5 via IMAP to access my work mail as well, but 
I learned that I better do not point it at that insane kernel-ml folder. 
Cause it seems that *even with its flat index files* it is not able to cope 
with it in any reasonable amount of time.


Now lets search for your name in *all* of my mails. How long will this 
take?

Granted this took about 20 seconds. But now I know that you have been 
replied to regarding some patch about

[PATCH 2/2] Input: tsc2007 - Add a z1_low_threshhold platform data 
parameter

posted on linux-input and linux-kernel mailinglists as well as I see this 
thread. As well as other occurences where you seem to be involved with 
Linux kernel development.

Granted when I now scroll down it takes some more seconds to extend the 
search result lists. So searching takes *some* time, but we are talking 
about a index of several hundred thousands or even millions of mails since 
co-workers have mail accounts, too. Also granted is that here Lucene is 
involved too.


Dmitry, can we agree from that that it is not the database that 
necessarily forces accessing or storing mails to be slow?

Yes, the server is a VM with 12 GB of RAM - last time I looked about 4 GB 
of it not even used for caches - on a fast server machine with several 
CPUs. But then my ThinkPad T520 with Dualcore i5 Sandybridge, Intel SSD 
320 and 8 GB of RAM isn´t that slow either and I expect that a well done 
database based Akonadi with a well done Nepomuk index should be able to 
deliver a comparable performance. Especially as I do not have to share the 
computing and I/O performance of the laptop with anybody else.

If Akonadi / KMail 2 does not deliver a similar performance yet, it sure 
needs optimization, but my bet clearly is that it is not the database. 
*Not at all*.

I do not know at the moment whether Zimbra uses more than one database, 
but actually I don´t see why it should and I can look it up next week. I 
don´t know the table layout either. Frankly I never cared up to now, cause 
I have never seen anything that lets me access my mails that fast. Even 
the IMAP server subjectively easily feels as fast or faster than Dovecot.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/