[Nepomuk] Nepomuk::ResourceWatcher: Performance issues

[Vishesh Handa]

On Fri, Apr 6, 2012 at 6:18 PM, Peter Penz <peter.penz19 at gmail.com> wrote:

> Hi,
>

Hey Peter


>
> As mentioned in my mail 2 days ago I'm using Nepomuk's ResourceWatcher to
> be able to reflect resource/property-changes in Dolphin's view.
>
> It works quite fine already but I observed a few implementation issues in
> ResourceWatcher that affect the performance in a bad way... I guess it
> should be no big deal to fix them, however please let me explain first how
> I use the ResourceWatcher in Dolphin:
>
> Dolphin resolves meta-data of directory-items step by step to prevent that
> the user-interface gets blocked. So when opening a directory with lets say
> 20000 items the ResourceWatcher gets used like this:
>
> 1. After receiving the 20000 items from KDirLister an instance of
> ResourceWatcher is created and ResourceWatcher::start() is invoked.
>
> 2. Asynchronously resolve the metadata of item 1:
>   - Read the meta-data and apply it to the Dolphin-model
>   - Add item 1 as resource to the ResourceWatcher
> (ResourceWatcher::addResource(**)).
>
> 3. When entering the next event-loop resolve the metadata of item 2
>   ...
>
> 4. Resolve the metadata of item 3
>   ...
>
> 5. ... and so on (20000 times)
>
> Now to the problems:
>
> - Invoking ResourceWatcher::start() in step 1 without added resources
> blocks the process (at least in my environment) for a few seconds. The
> problem is gone if I invoke start() after at least one resource has been
> added.
>

>From what I remember, the dbus call is blocking. It really shouldn't be.


> - Invoking ResourceWatcher::addResource() after invoking
> ResourceWatcher::start() has no effect. As workaround before adding a
> resource I needed to invoke ResourceWatcher::stop(), add the resource and
> invoke ResourceWatcher::start() again. The problem with this workaround is
> the performance: Calling stop() clears e.g. 15000 items and calling start()
> after adding one resource adds 15001 items again. In the case of the
> directory with 20000 this might take up a few seconds for each (!) of the
> items in the upper-range.
>

I see.

Btw, from a performance perspective, it would be better if you only
monitored the resources and properties you're interested in. In your case
that would be nao:numericRating and nao:hasTag.


>
> Please note that calling ResourceWatcher::start() after resolving the
> metadata of the 20000 items is way too late, the data can already be
> changed during the resolving. Also adding 20000 resources in one step takes
> around 110 seconds here :-/
>
> From a quick look at the implementation I guess it should not be too
> tricky to let the ResourceWatcher internally be aware whether it has been
> started already or not and to consider this when adding a new resource
> without doing an expensive stop/start call.
>
> It would be great if this could be fixed by the Nepomuk-team but I'm aware
> that your time is limited. So my question: Should I submit a bug-report and
> is it realistic that you can fix this before 4.9.0? Or should I try provide
> a patch?  My time is also limited and I'd prefer the
> "Nepomuk-team-fixes-it"-**option, but without a fix this would be a kind
> of showstopper from a Dolphin perspective and... - well, I'd provide a
> patch if you cannot work on this ;-)
>
> Still crossing my fingers that you say "no big deal, easy fixable"...
>

I'll try it out and look at it.

I'm a little busy till Friday, so I'll probably look at it over the weekend.


>
> Thanks,
> Peter
>
> ______________________________**_________________
> Nepomuk mailing list
> Nepomuk at kde.org
> https://mail.kde.org/mailman/**listinfo/nepomuk<https://mail.kde.org/mailman/listinfo/nepomuk>
>



-- 
Vishesh Handa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20120409/2c571a2d/attachment.html>