[Nepomuk] Nepomuk in 4.13 and beyond

[Vishesh Handa]

Hey everyone

During the KDE 4.11 cycle Nepomuk reached a maturity level that we were 
happy 
with, it is reasonably fast, stable, and unless used together with Akonadi it 
is no longer the "CPU consumer" it was before. We reached this state after 
years of analyzing what was wrong and what could be improved to the 
point 
where we no longer think any more improvement is possible only by 
modifying 
our code.

The next place where we could seek improvement was the RDF storage. We 
have 
been using Virtuoso for about 4 years and it's been a game changer for us 
performing way faster than any other of the alternatives we ever used 
before 
and more efficiently, but as many of you know (and others suffer) it is not a 
RDF 
storage designed for the desktop and it will never be. Since nothing better 
than Virtuoso exists for our use-case, we started to implement our own 
RDF storage mechanism (codenamed Vishuoso).

At some point during that progress we took a step back and re-analyzed 
the 
problems of the workspace and the current implementation. The problems 
are -

- Resource Description Framework (RDF)
The biggest problem with RDF is that it raises the knowledge needed to 
contribute to a point where most developers decide to to skip it. After all 
these years only a handful of brave developers have worked with it and the 
experience hasn't been good.

Then we need something easier to use so we can see a more broad 
adoption.

Additionally, RDF is a very flexible way to store data, it is however not the 
most efficient way. Data is generally completely normalized even though it 
is 
quite often not required. Eg - One doesn't need to store music file artists 
as 
a separate contact. This is great, from a theoretical point of view, but it is 
not very useful in practice.

- RDF Storage
There is no existing RDF storage designed to work in a Desktop. Virtuoso 
is 
great but it quickly uses hundreds of megabytes of ram and it has its own 
share of problems. The other alternative is tracker, but they lack certain 
features required in Nepomuk.

- Data duplication
Nepomuk has been used as both a search store and a data store. This 
results in 
massive data duplication and synchronization problems. In the case of 
Akonadi, 
emails are stored in Akonadi and are then duplicated in virtuoso, and are 
then 
duplicated in virtuoso's index. Every time data is changed in Akonadi it has 
to be updated in Nepomuk and vice-versa. This results in a process being 
responsible just for synchronizing the two stores.

- API Duplication
With the data residing in both Nepomuk and other stores 
(Akonadi/Files/etc), 
it isn't always clear which store it should be accessed from. This 
essentially 
results in duplication of APIs. Eg - Using KABC vs accessing contacts from 
Nepomuk.

These problems would still exist even if we had the fastest and most 
efficient 
RDF storage in the world.

At this point it was clear to us that the future was not going to be RDF. The 
next thing we did was to analyze our actual needs based on the last 5 
years of 
Nepomuk.

Our needs are -
* Full text index for searching
* Data store for simple objects such as tags / ratings / activities / etc
* Relations - Forming relations between different objects. Eg - This "file" is 
related to this "activity" or "person".
 
Each of these problems is independently solvable without RDF.

About 2 months ago we started to draft Baloo [1], a metadata solution 
that 
will cover the bare necessities of each use case we have. 

I'd like to avoid getting into the technical details of the implementation in 
this thread. Another thread can be started about its different aspects 
once 
you've read the basic ideas behind Baloo [2]

Current Plans
---------------------

After a month of designing the solution and a month of implementing it, 
Baloo 
is working way better than Nepomuk does. So, I'd like to switch to Baloo by 
default in 4.13, while keeping Nepomuk in maintenance mode for more 
conservative distributions.

This is not a completely new project as large parts of Baloo code are 
derived from Nepomuk and therefore comes with years of testing and real 
world use.
 
Baloo was also discussed in PIM Sprint and the PIM developers are happy 
to 
completely drop Nepomuk support for 4.13 and move to Baloo. Similarly, 
the 
telepathy developers are also working on moving KPeople away from 
Nepomuk.

Migration - There will be an automated migration mechanism for migrating 
tags, 
ratings and comments from Nepomuk to Baloo.

Trying it out?
-------------------

Developers are welcome to try out Baloo and have a look at the current 
source 
code. It's a still a work in progress, but we strongly feel that it is a step 
in the right direction.

I'd recommend using Milou [3] for searching.

-- 
Vishesh Handa

[1] https://projects.kde.org/projects/playground/base/baloo
[2] http://techbase.kde.org/Projects/Baloo
[3] https://projects.kde.org/projects/playground/base/milou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20131212/84e40585/attachment-0001.html>