[Nepomuk] [RFC] Avoid communicating through the Nepomuk Storage

[Ignacio Serantes]

Sounds good for me.


On Sun, May 26, 2013 at 12:58 AM, Vishesh Handa <me at vhanda.in> wrote:

> Hey guys
>
> I have made a very important discovery - The Storage service is a big
> bottleneck!
>
> Running a query such as - 'select * where { graph ?g { ?r ?p ?o. } } LIMIT
> 50000' by directly connecting to virtuoso via ODBC takes about 2.65
> seconds. In contrast running the same query by using the Nepomuk
> ResourceManager's main model takes about 19.5 seconds.
>
> Nepomuk internally uses the Soprano::LocalSocketClient to connect to the
> storage service which runs a Soprano::LocalServer.
>
> I've been trying to optimize this Soprano code for some time now and from
> 4.9 we have a good 200% performance increase. But we can increase it a LOT
> more by just directly communicating with virtuoso.
>
> Pros -
> * 6-8x performance upgrade
> * The storage service isn't using such high cpu when reading
> * Accurate reporting - Suppose app 'x' does a costly query which requires
> a large number of results, then 'x' will have high cpu consumption.
> Currently both NepomukStorage and 'x' have very high cpu consumption.
>
> Cons -
> * Less Control - By having all queries go through the Nepomuk Storage we
> could theoretical build amazing tools to tell us which query is executing
> and how long it is taking. However, no such tool has ever been written - so
> we won't be loosing anything.
>
> Before 4.10 this could never have been done because we used to have a lot
> of code in the storage service which handled removable media and other
> devices. This code would often modify the sparql queries and modify the
> results. With 4.10, I threw away all that code.
>
> Comments?
>
> PS: This is only for read only operations. All writes should still go
> through the storage service. Though maybe we want to change that as well?
>
> --
> Vishesh Handa
>
> _______________________________________________
> Nepomuk mailing list
> Nepomuk at kde.org
> https://mail.kde.org/mailman/listinfo/nepomuk
>
>


-- 
Best wishes,
Ignacio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20130527/2a8c72ef/attachment.html>