[Kde-pim] Akonadi: single database design mistake?

[Dmitry Torokhov]

Hi Martin,

On Thu, Dec 01, 2011 at 06:50:23PM +0100, Martin Steigerwald wrote:
> Hi Dmitry!
> 
> I rarely post here as a user of KDEPIM, but this I like to comment:
> 
> Am Dienstag, 29. November 2011 schrieb Dmitry Torokhov:
> > > Another thing: If we ever come to the point where MySQL is the
> > > bottleneck, the  current architecture should make it rather simple
> > > to come up with an alternative, optimized architecture. Personally,
> > > I just doubt that we are able to design a relational database from
> > > scratch that will outperform MySQL so easily...
> > 
> > Thet is the main question: do we really want a monolithic database
> > here.
> 
> At work we use a Zimbra Collaboration Suite server as our groupware 
> solution. It uses MySQL to store mail metadata, Lucene to provide a search 
> index and files to store the actual mail. It is serving about 30 users all 
> day via Zimbra webclient, outlook and various IMAP clients - I partly use 
> KMail with it.
> 
> Now consider this:
> 
> - about 100 folders including subfolders
> - hundred of thousands of mails
> - several gigabytes of mail easily (do not see it in the webclient and do 
> not want to open up a VPN to look in the administrative interface for the 
> size right now)
> - folders with tens of thousands of mails

Here at work we like to eat our dogfood and so we also converted almost
entirely to Zimbra. So it is several thousand employees with so many
mails, server farm running Zimbra that is partitioned properly, etc,
etc. And it all scaled and partitioned nicely and if our infrastructure
guys see that the load on one of the nodes gets too big they can bring
in another node and so forth. So yes, MySQL apparently can handle that.
I never said that MySQL is not suitable for large amounts of data.

But the point I was trying to make is that I do not want to replicate
that setup on my tiny laptop. There is a reason I have IMAP - I
_offload_ tasks from laptop to other boxes, such as receiving and
sorting mail, anti-spam and anti-virus checks, etc, etc, so that the
laptop only does fraction of work required. I do not want to fine-tune
MySQL on laptop to make sure indices fit into memory, that the log size
is appropriate, and so forth. And my question was - given that there
normally a single user (as in person) working with a single folder at a
given time, would not it be more effective to restrict the size of the
data we are working with to that single folder instead of trying to
handle the data as whole.

To be fair, after the pain of initial import and after running for a
couple days, the system has settled down and is now usable.

Thanks.

-- 
Dmitry

P.S. No, I still haven't profiled kmail, busy with input stuff for
now...

_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/