Akonadi dsaster

[Daniel Vrátil]

On Friday, October 14, 2016 11:30:37 AM CEST Michael Mol wrote:
> On Friday, October 14, 2016 09:12:10 AM Daniel Vrátil wrote:
> > On Monday, October 10, 2016 10:12:42 AM CEST Michael Mol wrote:
> > > A *huge* chunk of this, for me, turned out to be in the abysmal MySQL
> > > configuration it defaults to. I posted about it on Google+ and on /r/kde
> > > several months ago.
> > > https://plus.google.com/+MichaelMolG/posts/HQQH6RHhLNw
> > > 
> > > I'll note my understanding of Akonadi has improved slightly since I
> > > wrote
> > > that; I wasn't able to find where Akonadi's MySQL configuration was
> > > kept,
> > > so I switched out to using a system mysqld.That's not necessary if you
> > > know where Akonadi sources its MySQL configuration from.
> > > 
> > > I'll also add that I'm syncing nearly a decade's worth of email from
> > > GMail
> > > into Akonadi, hundreds of thousands of emails. People without as much
> > > email
> > > history likely won't see the same kind of performance difficulties I do.
> > > 
> > > Here's the relevant excerpt:
> > > 
> > > Altogether, here are the changes I've made:
> > > 
> > > In my.cnf
> > 
> > <snip>
<snip>
> 
> Really, you should look at per-collection table partitioning. This would
> massively speed up Akonadi where it deals with a large collection at the
> same time as any other collection; even if your indexes are effective, they
> have to contemplate all the data in a table, which means a lookup on a tiny
> collection will be impacted by sharing the index with all the other
> collections in the table. Per-collection partitioning would mean each
> collection gets an index dedicated to it, which means hot collections will
> tend to have their index already present in the buffer pool.

That is certainly an interesting idea. 

How does that scale with, say hundreds of Collections, meaning hundreds of 
tables?

There would be a major problem with the so called virtual Collections, i.e. 
Collections that do not contain any Items on their own, but they have Items 
from other collections "linked" to them - e.g. the "Search" Collections . For 
that there's CollectionPimItemRelation table which makes the queries for "give 
me all items linked to Collection X" rather simple. But with per-Collection 
partitioning would require doing such query virtually accross all the tables 
at once. 

> 
> > I would like to see some sort of a self-balancing configuration, where the
> > database could "suggest" optimal settings based on long-time load and
> > available hardware resources. That would magically solve the issue
> > described above, but that's more of a day-dreaming :-)
> 
> That'd be nice if peoples' usage patterns were constant, but power users'
> tend not to be, IME.
> 
> My preference would be for there to be some sliders hidden away under an
> "Advanced/Performance" tab for configuring Akonadi. You might look at how
> scripts like mysqltuner.pl work, and see about using the logic you find
> there to come up with baseline recommended settings.

Indeed this is not the first time an "Advanced" configuration with sliders to 
tune some MySQL options was suggested.

Could be a nice junior-job ;-)

> 
> > Pablo made a good point elsewhere in this thread about the ridiculous
> > amount of queries we do. I've added some more optimizations to master
> > recently to speed up some things, but there's still a long way to go :(
> 
> Yeah, I'd have to learn a bit more about Akonadi's architecture, but I get a
> strong impression there's a lot you could do with batching.

Indeed. Unfortunately the architecture makes batching very difficult in certain 
cases (mostly in the cases where we would benefit from it the most), which is 
why it did not happen yet.

I can go into deeper regarding some of those aspects, I'll drop you and Pablo 
an email regarding this.


Dan

-- 
Daniel Vrátil
www.dvratil.cz | dvratil at kde.org
IRC: dvratil on Freenode (#kde, #kontact, #akonadi, #fedora-kde)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdepim-users/attachments/20161016/fa744cb9/attachment.sig>