[Kde-pim] Nepomuk feeder queuing problem

[Christian Mollekopf]

Hey Sebastian,

I don't see the problem with this as this timer just kickstarts the
processing and can be called as many times as you want. The ItemQueue
does the actual feeding of the data. The processing of the ItemQueue is
triggered by the processItem() function which simply does nothing if
called repeatedly (mRunningJobs > 0). processItem() is also the function
which gets called if you call setOnline() or addItem(), but that
shouldn't hurt as far as I can see.

So there should be at maximum two nepomuk jobs running at a time (one
per ItemQueue).

Are you sure virtuoso is really going crazy? While indexing virtuoso
takes all the cpu power it can get, but that seems normal to me. Also
the indexing can take veeery long, so it's quite possible that users
think virtuoso just went crazy.

Note that some merges of indexed items still fail (I believe it has to
do with the same email address appearing several times in an email),
maybe this could trigger a problem?
I could send you a testitem which triggers the problem in case you wish
(it contains private data and is therefore not suitable to be uploaded
on a bugtracker).

Also it seems that some items get regularly reindexed because some
property of an akonadi-item changed (which always triggers a full
reindexing). I'm not yet sure why that happens exactly though. This just
to say that the feeder can produce similar load to the initial indexing
also after the initial indexing has finished.

So I think there are lots of improvements to be made regarding
performance, but I'm not aware of a problem that overloads nepomuk.

Thanks for looking into this.

Cheers,
Christian

On Mon, Jan 16, 2012, at 06:01 PM, Sebastian Trüg wrote:
> Hi Christian,
> 
> there seems to be a problem with the Nepomuk feeder in KDEpim. The
> reports about virtuoso going crazy pile up and now I ran into the same
> problem.
> Thus, I had a look at the feeder and found a potential problem:
> 
> You use a single shot timer to continue the indexing. This timer tells
> the queue to continue the processing of the items. This is all fine
> internally. But both setOnline() and addItem() start the timer without
> checking if any job has already been started. Thus, another job will be
> started. This can potentially lead to tons of nepomuk jobs running which
> could in fact make virtuoso go crazy as there is no real protection
> against such an "attack" in Nepomuk.
> 
> Please have a look and tell me if I read the code correctly.
> 
> This is something we should sort out before the final tag.
> 
> Cheers,
> Sebastian
> 
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/