[Kde-pim] Nepomuk feeder queuing problem

[Christian Mollekopf]

On Tuesday 17 January 2012 17.37:19 Michael Jansen wrote:
> On Monday, January 16, 2012 06:41:33 PM Christian Mollekopf wrote:
> > Hey Sebastian,
> > 
> > I don't see the problem with this as this timer just kickstarts the
> > processing and can be called as many times as you want. The ItemQueue
> > does the actual feeding of the data. The processing of the ItemQueue is
> > triggered by the processItem() function which simply does nothing if
> > called repeatedly (mRunningJobs > 0). processItem() is also the function
> > which gets called if you call setOnline() or addItem(), but that
> > shouldn't hurt as far as I can see.
> > 
> > So there should be at maximum two nepomuk jobs running at a time (one
> > per ItemQueue).
> > 
> > Are you sure virtuoso is really going crazy? While indexing virtuoso
> > takes all the cpu power it can get, but that seems normal to me. Also
> > the indexing can take veeery long, so it's quite possible that users
> > think virtuoso just went crazy.
> 
> No. I am sometimes sitting besides my desktop working on my laptop and enjoy
> virtuoso going crazy and stopping by just listening into the fan. So i put
> a top up and glimpsed over to see what happens.
> 
> It will go to 80-180% cpu for some time, then go back to normal, nearly all
> of the time. It never stops doing that here. Usually i was able to find out
>  the culprit by just looking at top. Another process was usually right
> behind virtuoso in cpu usage.
> 
> We fixed most of these problems.
> 
> This time it is different. Only virtuoso does it. Some akonadi process seem
> to hover at 3-5% for me but not really sure if there is any relation.
> 
> I have no clue if it is related. But on session login i enjoy virtuoso,
> nepomukstorage, kontact and akonadi_nepomukfeeder go onto a minutes long cpu
> and io burning session. I plan to have a look during the next days what is
> the reason for that.
> 

That virtuoso peaks at session start is the same here. However after a minute 
or so everything goes back to normal. 

Note that the feeder doesn't even really use a lot of cpu here when its full 
speed indexing, the load is really generated on virtuoso. You should however 
see if the feeder is active if you open up akonadiconsole (it would print 
which collection it is currently indexing as status message).

Cheers,
Christian

> Mike
> 
> > Note that some merges of indexed items still fail (I believe it has to
> > do with the same email address appearing several times in an email),
> > maybe this could trigger a problem?
> > I could send you a testitem which triggers the problem in case you wish
> > (it contains private data and is therefore not suitable to be uploaded
> > on a bugtracker).
> > 
> > Also it seems that some items get regularly reindexed because some
> > property of an akonadi-item changed (which always triggers a full
> > reindexing). I'm not yet sure why that happens exactly though. This just
> > to say that the feeder can produce similar load to the initial indexing
> > also after the initial indexing has finished.
> > 
> > So I think there are lots of improvements to be made regarding
> > performance, but I'm not aware of a problem that overloads nepomuk.
> > 
> > Thanks for looking into this.
> > 
> > Cheers,
> > Christian
> > 
> > On Mon, Jan 16, 2012, at 06:01 PM, Sebastian Trüg wrote:
> > > Hi Christian,
> > > 
> > > there seems to be a problem with the Nepomuk feeder in KDEpim. The
> > > reports about virtuoso going crazy pile up and now I ran into the same
> > > problem.
> > > Thus, I had a look at the feeder and found a potential problem:
> > > 
> > > You use a single shot timer to continue the indexing. This timer tells
> > > the queue to continue the processing of the items. This is all fine
> > > internally. But both setOnline() and addItem() start the timer without
> > > checking if any job has already been started. Thus, another job will be
> > > started. This can potentially lead to tons of nepomuk jobs running which
> > > could in fact make virtuoso go crazy as there is no real protection
> > > against such an "attack" in Nepomuk.
> > > 
> > > Please have a look and tell me if I read the code correctly.
> > > 
> > > This is something we should sort out before the final tag.
> > > 
> > > Cheers,
> > > Sebastian
> > 
> > _______________________________________________
> > KDE PIM mailing list kde-pim at kde.org
> > https://mail.kde.org/mailman/listinfo/kde-pim
> > KDE PIM home page at http://pim.kde.org/
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/