[Nepomuk] [Kde-pim] Nepomuk feeder queuing problem

[Allen Winter]

On Monday 16 January 2012 12:41:33 PM Christian Mollekopf wrote:
> Hey Sebastian,
> 
> I don't see the problem with this as this timer just kickstarts the
> processing and can be called as many times as you want. The ItemQueue
> does the actual feeding of the data. The processing of the ItemQueue is
> triggered by the processItem() function which simply does nothing if
> called repeatedly (mRunningJobs > 0). processItem() is also the function
> which gets called if you call setOnline() or addItem(), but that
> shouldn't hurt as far as I can see.
> 
> So there should be at maximum two nepomuk jobs running at a time (one
> per ItemQueue).
> 
> Are you sure virtuoso is really going crazy? While indexing virtuoso
> takes all the cpu power it can get, but that seems normal to me. Also
> the indexing can take veeery long, so it's quite possible that users
> think virtuoso just went crazy.
> 
> Note that some merges of indexed items still fail (I believe it has to
> do with the same email address appearing several times in an email),
> maybe this could trigger a problem?
> I could send you a testitem which triggers the problem in case you wish
> (it contains private data and is therefore not suitable to be uploaded
> on a bugtracker).
> 
> Also it seems that some items get regularly reindexed because some
> property of an akonadi-item changed (which always triggers a full
> reindexing). I'm not yet sure why that happens exactly though. This just
> to say that the feeder can produce similar load to the initial indexing
> also after the initial indexing has finished.
> 
> So I think there are lots of improvements to be made regarding
> performance, but I'm not aware of a problem that overloads nepomuk.
> 
> Thanks for looking into this.
> 
> Cheers,
> Christian
> 
> On Mon, Jan 16, 2012, at 06:01 PM, Sebastian Trüg wrote:
> > Hi Christian,
> > 
> > there seems to be a problem with the Nepomuk feeder in KDEpim. The
> > reports about virtuoso going crazy pile up and now I ran into the same
> > problem.
> > Thus, I had a look at the feeder and found a potential problem:
> > 
> > You use a single shot timer to continue the indexing. This timer tells
> > the queue to continue the processing of the items. This is all fine
> > internally. But both setOnline() and addItem() start the timer without
> > checking if any job has already been started. Thus, another job will be
> > started. This can potentially lead to tons of nepomuk jobs running which
> > could in fact make virtuoso go crazy as there is no real protection
> > against such an "attack" in Nepomuk.
> > 
> > Please have a look and tell me if I read the code correctly.
> > 
> > This is something we should sort out before the final tag.

Also please try to find some time to fix the searching problems identified
by Volker and Christian in the thread "Does KMail Searching Work for Anyone?"

Really something else we need to have fixed before 4.8.0 final.