[Kde-pim] Re: Architecture: Akonadi, Nepomuk, SPARQL

[Volker Krause]

On Tuesday 26 April 2011 16:21:36 Michael Schuerig wrote:
> On Monday 25 April 2011, Volker Krause wrote:
> > On Friday 22 April 2011 23:41:13 Michael Schuerig wrote:
> Volker,
> 
> thanks for your illuminating response.
> 
> [...]
> 
> > > There are a few things about this picture that I don't understand,
> > > assuming I've got the picture correctly. The first thing is why
> > > there need to be two separate database servers when Virtuoso
> > > itself is a quite capable relational/SQL database? Storing related
> > > data consistently in two databases instead of just one surely
> > > doesn't make things easier.
> > 
> > It started this way due to historical reasons, Nepomuk wasn't using
> > Virtuoso back then for example. During Akademy last year we tried to
> > get Akonadi working with Virtuoso as its database backend. It turned
> > out though that it's not that easy, the SQL dialects are too
> > different. We got to the point where Akonadi was able to connect to
> > Virtuoso and with some hacks even create its tables but still failed
> > when trying to write its data. Nothing really hard, but probably a
> > week or two of work that needs to be done.
> 
> I'm surprised by this, as I didn't expect that Akonadi needs anything
> beyond basic SQL, but I'll take your word for it. Is the code resulting
> from this experiment somewhere in the repos?

Akonadi indeed doesn't use more than basic SQL, but even that differs slightly 
between the various databases. The first one you usually run into is different 
type names (TIMESTAMP vs. DATETIME, BLOB vs BYTEA), but there are also more 
subtle ones, like difference in unicode string handling. With Virtuoso we 
additional run into problems that some of our column names are apparently 
keywords there (e.g. "external"), causing some new problems.

Anyway, all the code is in Git, look for the ODBC backend (that's how we 
connect to Virtuoso via QtSQL). The only thing that wasn't committed IIRC was 
the column renaming hack we did, since that would break existing installations 
and would need a more sophisticated solution. I might still have that 
somewhere, so if you are interested I can look for it.

> > However, just using Virtuoso this way will not magically solve
> > anything, we need both ways, bit-perfect retrieval of the payload
> > and (full text/semantic) indexing. Inherently, this will roughly
> > duplicate the amount of stored data, no matter if you use one or two
> > databases (in theory, in practice using the same database can of
> > course have some synergies in resource usage etc, that's why we
> > tried it).
> 
> I agree that the "blob" (opaque Akonadi payload data) is necessary. What
> concerns me about MySQL + Virtuoso is not the on-disk footprint. Even
> the memory consumption for the MySQL process isn't that bad. Having two
> separate servers leads to new failure modes: e.g. when only one of them
> is running, when data becomes inconsistent. On a single server you can
> have guaranteed foreign-key relationships, between servers its much more
> effort to follow associations.

Right, but this would also require Nepomuk and Akonadi to become mutally 
dependent on each other. Even with the Virtuoso backend work for Akonadi, it 
would just use another database in Virtuoso, not the same one as Nepomuk. So, 
this would still not address your concern.

Making them mutually dependent is problematic though, Akonadi is also used on 
constraint mobile devices where Nepomuk is not (yet) available, while Nepomuk 
uses all kinds of different data sources and is in no way limited to Akonadi 
(nor would that be desired).

However, the current situation is not as bad as it looks like, since Akonadi 
has a change tracking/change replay mechanism designed for keeping stuff in 
sync (which is one of its main usecases after all), this is also used for 
feeding data into Nepomuk.

regards
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kde-pim/attachments/20110429/e4c81af7/attachment.sig>
-------------- next part --------------
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/