[Kde-pim] Nepomuk feeder queuing problem

[Sebastian Trüg]

On 01/18/2012 10:04 AM, Christian Mollekopf wrote:
> On Wednesday 18 January 2012 09.14:31 Sebastian =?ISO-8859-1?Q?Tr=FCg?= wrote:
>> On 01/17/2012 12:14 AM, Allen Winter wrote:
>>> On Monday 16 January 2012 12:41:33 PM Christian Mollekopf wrote:
>>>> Hey Sebastian,
>>>>
>>>> I don't see the problem with this as this timer just kickstarts the
>>>> processing and can be called as many times as you want. The ItemQueue
>>>> does the actual feeding of the data. The processing of the ItemQueue is
>>>> triggered by the processItem() function which simply does nothing if
>>>> called repeatedly (mRunningJobs > 0). processItem() is also the function
>>>> which gets called if you call setOnline() or addItem(), but that
>>>> shouldn't hurt as far as I can see.
>>>>
>>>> So there should be at maximum two nepomuk jobs running at a time (one
>>>> per ItemQueue).
>>>>
>>>> Are you sure virtuoso is really going crazy? While indexing virtuoso
>>>> takes all the cpu power it can get, but that seems normal to me. Also
>>>> the indexing can take veeery long, so it's quite possible that users
>>>> think virtuoso just went crazy.
>>>>
>>>> Note that some merges of indexed items still fail (I believe it has to
>>>> do with the same email address appearing several times in an email),
>>>> maybe this could trigger a problem?
>>>> I could send you a testitem which triggers the problem in case you wish
>>>> (it contains private data and is therefore not suitable to be uploaded
>>>> on a bugtracker).
>>>>
>>>> Also it seems that some items get regularly reindexed because some
>>>> property of an akonadi-item changed (which always triggers a full
>>>> reindexing). I'm not yet sure why that happens exactly though. This just
>>>> to say that the feeder can produce similar load to the initial indexing
>>>> also after the initial indexing has finished.
>>>>
>>>> So I think there are lots of improvements to be made regarding
>>>> performance, but I'm not aware of a problem that overloads nepomuk.
>>>>
>>>> Thanks for looking into this.
>>>>
>>>> Cheers,
>>>> Christian
>>>>
>>>> On Mon, Jan 16, 2012, at 06:01 PM, Sebastian Trüg wrote:
>>>>> Hi Christian,
>>>>>
>>>>> there seems to be a problem with the Nepomuk feeder in KDEpim. The
>>>>> reports about virtuoso going crazy pile up and now I ran into the same
>>>>> problem.
>>>>> Thus, I had a look at the feeder and found a potential problem:
>>>>>
>>>>> You use a single shot timer to continue the indexing. This timer tells
>>>>> the queue to continue the processing of the items. This is all fine
>>>>> internally. But both setOnline() and addItem() start the timer without
>>>>> checking if any job has already been started. Thus, another job will be
>>>>> started. This can potentially lead to tons of nepomuk jobs running which
>>>>> could in fact make virtuoso go crazy as there is no real protection
>>>>> against such an "attack" in Nepomuk.
>>>>>
>>>>> Please have a look and tell me if I read the code correctly.
>>>>>
>>>>> This is something we should sort out before the final tag.
>>>
>>> Also please try to find some time to fix the searching problems identified
>>> by Volker and Christian in the thread "Does KMail Searching Work for
>>> Anyone?"
>> I did not find anything. Could you please direct me to the problem.
> 
> The problem lies in the akonadi server: 
> akonadi/server/src/search/nepomuksearchengine.cpp
> 
> The searching works (the correct result is found), but akonadi still expects 
> an akonadi id from nepomuk. I tried fixing this using the, in kmail already 
> requested, http://akonadi-project.org/ontologies/aneo#akonadiItemId property.
> 
> This requested property never makes it to the server however. I suspect that 
> it is because the dbus interface changed, because there is yet another copy of 
> that nepomuk code in the akonadi server.

true, request properties are transferred in a different way. They are
essentially ComparisonTerm variableNames with a dedicated prefix. It is
a bit hacky.

> The actual query is built in kdepim/mailcommon/searchpattern.cpp (in 
> asSparqlQuery()), but I don't think it's a problem of the query.
> 
> Any help with fixing the dbus interface is greatly appreciated as I failed so 
> far to fix it. Simply updating the interface didn't work because I'd have to 
> copy almost complete libnepomukcore. We really need that separate nepomuk 
> repository....

Actually I implemented backwards-compatibility a long while ago but
never got any feedback on it from the kde-pim team. Thus, it just
gathers dust in branch nepomuk/legacy-result. I am not sure about the
state of it.

Cheers,
Sebastian

_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/