Nepomuk feature freeze No. 2

[Sebastian Trüg]

On Tuesday 18 September 2007 18:43:33 Richard Dale wrote:
> On Tuesday 18 September 2007, Sebastian Trüg wrote:
> > Hi again,
> >
> > after the last discussion I could not sleep well and that resulted in
> > Soprano now having three interfaces: TCP, Unix sockets, and DBus. The
> > performance results are not really what I expected and by far not as
> > impressive as I thought.
> >
> > Task                            | TCP   | Unix Socket | DBus |
> > --------------------------------------------------------------
> > adding one statement            |    1  |    1        |    2 |
> > adding list of 1000 statements  | 1151  | 1207        | 1343 |
> > adding 1000 statements          |  914  |  958        | 1080 |
> >
> > (all values in milliseconds)
> >
> > So DBus is slower but only marginal. What confuses me here is that the
> > TCP interface is faster than the unix socket one. That seems a bid odd.
>
> Do Unix sockets write to a disk file - is that why? I'm pleased to see that
> dbus is reasonably fast apart from slower latency, which is what I expected
> to see. So it looks like a bulk loading feature would be a good idea? When
> I run strigi it looks as though it is sending one message per triple,
> rather than bulking them up - is that right?

Strigi does not write any triples. It uses its own clucene index. The Soprano 
backend does only live here on my computer and does not work properly yet. :(

> > Anyway, I don't think TCP is the right solution for now as it would
> > require the introduction of permission handling. And seeing that unix
> > sockets and DBus (after all, DBus uses unix sockets, too) do not differ
> > that much it is probably best to stick with the well known, reliable DBus
> > interface. After all, it will make programming against it from other
> > languages much easier.
>
> Yes, I'm pleased about that - from the point of view of ruby bindings dbus
> will be much better that needing to wrap yet another app specific IPC api.
>
> Because I haven't actually written an app using the nepomuk daemon yet it
> is hard to imagine uses cases, but it seems to me that it would be useful
> to register for dbus signal that the nepomuk daemon would emit on, say when
> a new foaf:person was added to the datastore. Then the UI of your personal
> contacts dbus client or whatever could be updated to show the new person.

That sounds like a very nice idea. However, at the moment it is hard to 
implement as we have only generic statementsAdded signals. We would need 
signals that include the added triples however. I am not sure how performant 
a signal like "statementAdded( Statement )" can be...

> > Now, why do I write this mail at all if I want to stick with the DBus
> > interface. Well, at the moment Nepomuk in kdelibs uses the Nepomuk
> > middleware I implemented some month back when there were plans in the
> > Nepomuk project itself for a federation. That would have meant that
> > Nepomuk-KDE could have reused services implemented in other languages
> > (most likely java since everything in Nepomuk is hacked in java). Now
> > that the Nepomuk project dropped the idea of the federation the
> > middleware as it is in kdelibs only introduces unnecessary complexity
> > which I would like to remove (So far we only have one Nepomuk-KDE service
> > anyway: the RDF repository; and there is no reason to not use KDE service
> > technologies now.)
> > In its place I would like to use the new Soprano DBus interface. it has
> > the following advantages:
> >
> >   * Uses the clean and (IMHO) very nice Soprano 2 API
> >   * Is completely iterator-based which means that handling large
> > quantaties becomes cleaner.
> >   * We do not duplicate interfaces (Soprano + the one in kdelibs).
> >
> > Replacing the middleware would mean:
> >   * Updating Soprano in kdesupport to version 2 (currently in
> >     branches/work/soprano2)
> >   * removing kdelibs/nepomuk/middleware entirely
> >   * small internal changes to kdelibs/nepomuk/core to make it use Soprano
> >   * Change kdebase/runtime/nepomuk/coreservices to a plain KDE service
> >     without Nepomuk service registration and update it to Soprano 2
> >     including the new Soprano 2 DBus interface
> >
> > The public API would only change in one position (except for the removal
> > of the middleware which no one uses anyway AFAIK):
> >   * Nepomuk::ResouceManager::serviceRegistry() removed
> >
> > Well, that is it I think.
> > Your comments please.
>
> I assume we can still get SPARQL results in xml format as you described in
> your last mails? If so, it all sounds really good to me, and I'm looking

That is not implemented yet. The Soprano server does use specific structs for 
that but they are pretty easy to implement I think. Basically you have a Node 
which is an int and three strings and a statement which is four nodes. That 
is all.
But if you want XML we can always create an additional interface that provides 
a SPARQL endpoint returning XML data. I just did not put it into the Soprano 
interface as it seems like a client issue to me. But we can discuss that 
without a doubt. Maybe a good solution would be to extend the Soprano 
interface when using it in Nepomuk and simply adding another method which 
returns XML data.

> forward to trying it out. Where do the triples actually get stored -  there
> seem to be two triple stores, one in ~/.nepomuk/share/storage/system and
> another one too? The nepomuk daemon has never written anything to
> the .nepomuk one as far as I can see, although it does get created on start
> up.

The problem here is that redland (which is still used) does not sync back 
everything before being deleted. And the nepomuk daemon in svn does not catch 
signals yet, thus, when killed, does not properly sync to disk. I only 
realized that a short while back and my local version already does that 
properly.

Cheers,
Sebastian