[Nepomuk] Nepomuk::ResourceWatcher: Performance issues

[Sebastian Trüg]

On 04/06/2012 02:48 PM, Peter Penz wrote:
> Hi,
> 
> As mentioned in my mail 2 days ago I'm using Nepomuk's ResourceWatcher
> to be able to reflect resource/property-changes in Dolphin's view.
> 
> It works quite fine already but I observed a few implementation issues
> in ResourceWatcher that affect the performance in a bad way... I guess
> it should be no big deal to fix them, however please let me explain
> first how I use the ResourceWatcher in Dolphin:
> 
> Dolphin resolves meta-data of directory-items step by step to prevent
> that the user-interface gets blocked. So when opening a directory with
> lets say 20000 items the ResourceWatcher gets used like this:
> 
> 1. After receiving the 20000 items from KDirLister an instance of
> ResourceWatcher is created and ResourceWatcher::start() is invoked.
> 
> 2. Asynchronously resolve the metadata of item 1:
>    - Read the meta-data and apply it to the Dolphin-model
>    - Add item 1 as resource to the ResourceWatcher
> (ResourceWatcher::addResource()).
> 
> 3. When entering the next event-loop resolve the metadata of item 2
>    ...
> 
> 4. Resolve the metadata of item 3
>    ...
> 
> 5. ... and so on (20000 times)
> 
> Now to the problems:
> 
> - Invoking ResourceWatcher::start() in step 1 without added resources
> blocks the process (at least in my environment) for a few seconds. The
> problem is gone if I invoke start() after at least one resource has been
> added.

I might have a lot of un-pushed RW commits in some branch... I have so
many nepomuk branches with so many fixes and improvements with so many
unit tests missing....

> - Invoking ResourceWatcher::addResource() after invoking
> ResourceWatcher::start() has no effect. As workaround before adding a
> resource I needed to invoke ResourceWatcher::stop(), add the resource
> and invoke ResourceWatcher::start() again. The problem with this
> workaround is the performance: Calling stop() clears e.g. 15000 items
> and calling start() after adding one resource adds 15001 items again. In
> the case of the directory with 20000 this might take up a few seconds
> for each (!) of the items in the upper-range.

I implemented support for that a while ago. Where did you copy the RW from?

> Please note that calling ResourceWatcher::start() after resolving the
> metadata of the 20000 items is way too late, the data can already be
> changed during the resolving. Also adding 20000 resources in one step
> takes around 110 seconds here :-/
> 
> From a quick look at the implementation I guess it should not be too
> tricky to let the ResourceWatcher internally be aware whether it has
> been started already or not and to consider this when adding a new
> resource without doing an expensive stop/start call.
> 
> It would be great if this could be fixed by the Nepomuk-team but I'm
> aware that your time is limited. So my question: Should I submit a
> bug-report and is it realistic that you can fix this before 4.9.0? Or
> should I try provide a patch?  My time is also limited and I'd prefer
> the "Nepomuk-team-fixes-it"-option, but without a fix this would be a
> kind of showstopper from a Dolphin perspective and... - well, I'd
> provide a patch if you cannot work on this ;-)
> 
> Still crossing my fingers that you say "no big deal, easy fixable"...
> 
> Thanks,
> Peter
> 
> _______________________________________________
> Nepomuk mailing list
> Nepomuk at kde.org
> https://mail.kde.org/mailman/listinfo/nepomuk
>