[Nepomuk] [Kde-pim] Nepomuk feeder queuing problem

[Christian Mollekopf]

On Wed, Jan 18, 2012, at 11:29 AM, Sebastian Trüg wrote:
> On 01/18/2012 10:04 AM, Christian Mollekopf wrote:
> > On Wednesday 18 January 2012 09.14:31 Sebastian =?ISO-8859-1?Q?Tr=FCg?= wrote:
> >> On 01/17/2012 12:14 AM, Allen Winter wrote:
> >>> On Monday 16 January 2012 12:41:33 PM Christian Mollekopf wrote:
> >>>> Hey Sebastian,
> >>>>
> >>>> I don't see the problem with this as this timer just kickstarts the
> >>>> processing and can be called as many times as you want. The ItemQueue
> >>>> does the actual feeding of the data. The processing of the ItemQueue is
> >>>> triggered by the processItem() function which simply does nothing if
> >>>> called repeatedly (mRunningJobs > 0). processItem() is also the function
> >>>> which gets called if you call setOnline() or addItem(), but that
> >>>> shouldn't hurt as far as I can see.
> >>>>
> >>>> So there should be at maximum two nepomuk jobs running at a time (one
> >>>> per ItemQueue).
> >>>>
> >>>> Are you sure virtuoso is really going crazy? While indexing virtuoso
> >>>> takes all the cpu power it can get, but that seems normal to me. Also
> >>>> the indexing can take veeery long, so it's quite possible that users
> >>>> think virtuoso just went crazy.
> >>>>
> >>>> Note that some merges of indexed items still fail (I believe it has to
> >>>> do with the same email address appearing several times in an email),
> >>>> maybe this could trigger a problem?
> >>>> I could send you a testitem which triggers the problem in case you wish
> >>>> (it contains private data and is therefore not suitable to be uploaded
> >>>> on a bugtracker).
> >>>>
> >>>> Also it seems that some items get regularly reindexed because some
> >>>> property of an akonadi-item changed (which always triggers a full
> >>>> reindexing). I'm not yet sure why that happens exactly though. This just
> >>>> to say that the feeder can produce similar load to the initial indexing
> >>>> also after the initial indexing has finished.
> >>>>
> >>>> So I think there are lots of improvements to be made regarding
> >>>> performance, but I'm not aware of a problem that overloads nepomuk.
> >>>>
> >>>> Thanks for looking into this.
> >>>>
> >>>> Cheers,
> >>>> Christian
> >>>>
> >>>> On Mon, Jan 16, 2012, at 06:01 PM, Sebastian Trüg wrote:
> >>>>> Hi Christian,
> >>>>>
> >>>>> there seems to be a problem with the Nepomuk feeder in KDEpim. The
> >>>>> reports about virtuoso going crazy pile up and now I ran into the same
> >>>>> problem.
> >>>>> Thus, I had a look at the feeder and found a potential problem:
> >>>>>
> >>>>> You use a single shot timer to continue the indexing. This timer tells
> >>>>> the queue to continue the processing of the items. This is all fine
> >>>>> internally. But both setOnline() and addItem() start the timer without
> >>>>> checking if any job has already been started. Thus, another job will be
> >>>>> started. This can potentially lead to tons of nepomuk jobs running which
> >>>>> could in fact make virtuoso go crazy as there is no real protection
> >>>>> against such an "attack" in Nepomuk.
> >>>>>
> >>>>> Please have a look and tell me if I read the code correctly.
> >>>>>
> >>>>> This is something we should sort out before the final tag.
> >>>
> >>> Also please try to find some time to fix the searching problems identified
> >>> by Volker and Christian in the thread "Does KMail Searching Work for
> >>> Anyone?"
> >> I did not find anything. Could you please direct me to the problem.
> > 
> > The problem lies in the akonadi server: 
> > akonadi/server/src/search/nepomuksearchengine.cpp
> > 
> > The searching works (the correct result is found), but akonadi still expects 
> > an akonadi id from nepomuk. I tried fixing this using the, in kmail already 
> > requested, http://akonadi-project.org/ontologies/aneo#akonadiItemId property.
> > 
> > This requested property never makes it to the server however. I suspect that 
> > it is because the dbus interface changed, because there is yet another copy of 
> > that nepomuk code in the akonadi server.
> 
> true, request properties are transferred in a different way. They are
> essentially ComparisonTerm variableNames with a dedicated prefix. It is
> a bit hacky.
> 

I suppose you're referring to the "reqProp1" names which are required in
the sparql query for the results bindings?
That wouldn't be a problem if we could use the same codebase everywhere
(nepomuk-core).

Meanwhile I found part of the problem in the queryserviceclient.cpp copy
in the akonadi-server. It passed an always empty QHash<QString,QString>
as requestedPropertyMap.
Now that I corrected this I get back one result, with the correct
property url, but the supplied Soprano::Node is still invalid (so I
can't get the value of the property).
The dbusoperators look ok so I'm a bit lost. Note that QUrl's are used
in the akonadi-server copy instead of Nepomuk::Types::Property, but
since that seems to be for the serialization of the property uri, which
is transported okay, that shouldn't be the problem, right? Somehow the
Soprano::Node marshaling fails as far as I understand.

> > The actual query is built in kdepim/mailcommon/searchpattern.cpp (in 
> > asSparqlQuery()), but I don't think it's a problem of the query.
> > 
> > Any help with fixing the dbus interface is greatly appreciated as I failed so 
> > far to fix it. Simply updating the interface didn't work because I'd have to 
> > copy almost complete libnepomukcore. We really need that separate nepomuk 
> > repository....
> 
> Actually I implemented backwards-compatibility a long while ago but
> never got any feedback on it from the kde-pim team. Thus, it just
> gathers dust in branch nepomuk/legacy-result. I am not sure about the
> state of it.
> 

Where is it? Coldn't find it in runitem/kdelibs/nepomuk-core...
And what exactly does it implement? Support for the query mechanism used
in the akonadi server?

If it's not possible to get rid of the kdelibs/kde-runtime copies of
nepomuk for 4.9 then I suggest that we start using nepomuk-core anyways
for projects like kdepim.
Copying those interfaces and parts of nepomuk around is just a huge
waste of time, and having two copies of nepomuk is still better than not
knowing how many there are at all...

Cheers,
Christian

> Cheers,
> Sebastian
> 
>