[Kde-pim] Nepomuk feeder queuing problem

[Christian Mollekopf]

On Wednesday 18 January 2012 09.14:31 Sebastian =?ISO-8859-1?Q?Tr=FCg?= wrote:
> On 01/17/2012 12:14 AM, Allen Winter wrote:
> > On Monday 16 January 2012 12:41:33 PM Christian Mollekopf wrote:
> >> Hey Sebastian,
> >> 
> >> I don't see the problem with this as this timer just kickstarts the
> >> processing and can be called as many times as you want. The ItemQueue
> >> does the actual feeding of the data. The processing of the ItemQueue is
> >> triggered by the processItem() function which simply does nothing if
> >> called repeatedly (mRunningJobs > 0). processItem() is also the function
> >> which gets called if you call setOnline() or addItem(), but that
> >> shouldn't hurt as far as I can see.
> >> 
> >> So there should be at maximum two nepomuk jobs running at a time (one
> >> per ItemQueue).
> >> 
> >> Are you sure virtuoso is really going crazy? While indexing virtuoso
> >> takes all the cpu power it can get, but that seems normal to me. Also
> >> the indexing can take veeery long, so it's quite possible that users
> >> think virtuoso just went crazy.
> >> 
> >> Note that some merges of indexed items still fail (I believe it has to
> >> do with the same email address appearing several times in an email),
> >> maybe this could trigger a problem?
> >> I could send you a testitem which triggers the problem in case you wish
> >> (it contains private data and is therefore not suitable to be uploaded
> >> on a bugtracker).
> >> 
> >> Also it seems that some items get regularly reindexed because some
> >> property of an akonadi-item changed (which always triggers a full
> >> reindexing). I'm not yet sure why that happens exactly though. This just
> >> to say that the feeder can produce similar load to the initial indexing
> >> also after the initial indexing has finished.
> >> 
> >> So I think there are lots of improvements to be made regarding
> >> performance, but I'm not aware of a problem that overloads nepomuk.
> >> 
> >> Thanks for looking into this.
> >> 
> >> Cheers,
> >> Christian
> >> 
> >> On Mon, Jan 16, 2012, at 06:01 PM, Sebastian Trüg wrote:
> >>> Hi Christian,
> >>> 
> >>> there seems to be a problem with the Nepomuk feeder in KDEpim. The
> >>> reports about virtuoso going crazy pile up and now I ran into the same
> >>> problem.
> >>> Thus, I had a look at the feeder and found a potential problem:
> >>> 
> >>> You use a single shot timer to continue the indexing. This timer tells
> >>> the queue to continue the processing of the items. This is all fine
> >>> internally. But both setOnline() and addItem() start the timer without
> >>> checking if any job has already been started. Thus, another job will be
> >>> started. This can potentially lead to tons of nepomuk jobs running which
> >>> could in fact make virtuoso go crazy as there is no real protection
> >>> against such an "attack" in Nepomuk.
> >>> 
> >>> Please have a look and tell me if I read the code correctly.
> >>> 
> >>> This is something we should sort out before the final tag.
> > 
> > Also please try to find some time to fix the searching problems identified
> > by Volker and Christian in the thread "Does KMail Searching Work for
> > Anyone?"
> I did not find anything. Could you please direct me to the problem.

The problem lies in the akonadi server: 
akonadi/server/src/search/nepomuksearchengine.cpp

The searching works (the correct result is found), but akonadi still expects 
an akonadi id from nepomuk. I tried fixing this using the, in kmail already 
requested, http://akonadi-project.org/ontologies/aneo#akonadiItemId property.

This requested property never makes it to the server however. I suspect that 
it is because the dbus interface changed, because there is yet another copy of 
that nepomuk code in the akonadi server.

The actual query is built in kdepim/mailcommon/searchpattern.cpp (in 
asSparqlQuery()), but I don't think it's a problem of the query.

Any help with fixing the dbus interface is greatly appreciated as I failed so 
far to fix it. Simply updating the interface didn't work because I'd have to 
copy almost complete libnepomukcore. We really need that separate nepomuk 
repository....

Cheers,
Christian

> 
> > Really something else we need to have fixed before 4.8.0 final.
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/