[Kde-pim] Excessive amount of queries

[Milian Wolff]

On Sunday 07 December 2014 15:52:07 Aaron J. Seigo wrote:
> On Sunday, December 7, 2014 07.56:10 Pablo Sanchez wrote:
> > consultant perspective.  Sometimes people forget to ask those questions:
> >   do we really need 500,000 IOPS to support our application for a single
> 
> not IOPs. query round trips.
> 
> that's 500,000 individual "give me record X" queries from the client being
> fulfilled from a 50,000 row dataset with rows being a little over 2k in size
> each and whole rows being returned.
> 
> can MySQL post those kinds of numbers, client->server->client, on a
> commodity laptop?

I think you are missing the point, Aaron. The question should be: Why do we 
even need 500k query round trips per second for a mail application?

> saturating available disk IOPs is doable with most reasonable RDBMSs, but
> that number is largely uninteresting. it's the query turn around time. and
> no, we probably don't need 500k queries/s, but given that given each query
> takes on average 1/500,000th of a second it means we can get away with
> querying and not worrying so much about it because the latency is so low.

Independent of the actual storage mechanism, doing less will always be more 
performant than doing more. So thinking about the proper way to tackle this 
issue, to find a solution that does not require thousands of queries per 
second, would pay of way more than rewriting all of Akonadi. And would still 
pay of if you want to rewrite it eventually.

> ... I'm pretty sure we can't get those #s from MySQL.
> 
> ... and with the key/value store we are working with right now it gets even
> better: it memory maps results and the buffer structure we are using reads
> from memory mapped data segments so that there is close to zero copy for
> results. Forget about cross-process data transfer costs, there is as close
> to zero *in-process* transfer costs as well.
> 
> This is ignoring other benefits like lower memory usage, not being able to
> share identical-query datasets easily, not having to deal with schema
> updates, ... ... ... from a purely performance perspective, going to a
> key/value store offers improvements that are hard to beat, improvements
> that are likely to render things like maintaining a query result cache, and
> all the complexity that comes with such things, unnecessary.
> 
> Which comes back to one of the biggest challenges with Akonadi today: to use
> it effectively means ever increasing code complexity on top of it .. and
> that complexity means more maintenance cost and in some places of the code
> a "fossilization" where changes become increasingly difficult due to pre-
> existing complexity.

I'm not arguing against anything of that. Just keep in mind that using some 
faster storage mechanism won't free the developers from actually thinking 
about the performance implications of the code they are writing.

Bye

-- 
Milian Wolff
mail at milianw.de
http://milianw.de
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/