[Nepomuk] Redundant DBus communication

[Vishesh Handa]

On Sun, Jan 13, 2013 at 5:57 PM, David Faure <faure at kde.org> wrote:

> On Sunday 13 January 2013 17:12:15 Vishesh Handa wrote:
> > You're right. We can possibly remove the added and removed signals and
> > infer that data from the changed signal.
>
> Won't this require expensive lookups on the receiver side?
> (If not, then I agree.)
>
> The alternative would be to emit added without emitting changed right away
> --
> but I don't know the current design (= what happens with these signals).
>

I'll put up a patch tomorrow.


>
> > They'll each have their own. This has mostly been done cause it is really
> > hard to implement it otherwise.
> >
> > On the server side we have 3 hash tables which maps the following keys  -
> > * Resources
> > * Properties
> > * Types
> >
> > to each ResourceWatcher.
>
> Yes, and I wonder if it happens often that the exact same set of keys is
> requested by the same code in multiple applications.
> In kdepim, I suspect this might happen? Or maybe I don't really understand
> the
> context. It's worth checking, at least.
>
> Each ResourceWatcher can watch a number of different combinations of all
> of
> > these (They conditions are ANDed). Therefore it is just a lot simpler for
> > each for each ResourceWatcher to listen to their own dbus path, instead
> of
> > creating special dbus paths for the intersection of resource/properties
> and
> > types.
>
> Right. I'm not suggesting intersections, but equality tests.
>


I've grepped the entire kdepimlibs, kdepim and kdepim-runtime for
ResourceWatcher. I fond the following -

vlap:~/kde/src/kdepim $ a ResourceWatcher
messagelist/storagemodel.cpp
39:#include <Nepomuk2/ResourceWatcher>
122:  Nepomuk2::ResourceWatcher *watcher = new
Nepomuk2::ResourceWatcher(this);

mailcommon/filter/filtermanager.cpp
30:#include <Nepomuk2/ResourceWatcher>
122:  Nepomuk2::ResourceWatcher *watcher = new
Nepomuk2::ResourceWatcher(this);

kmail/tagactionmanager.cpp
39:#include <nepomuk2/resourcewatcher.h>
62:  Nepomuk2::ResourceWatcher* watcher = new
Nepomuk2::ResourceWatcher(this);

The FilterManager and TagActionManager are actually both just watching for
the type nao:Tag. They could indeed be merged together. This would have be
done on the server side. I'm not sure exactly how to do it, but it is
possible. I'll try.


>
> If the same very very old email of mine is watched by more than one
> process,
> I'm pretty sure it's not related to my own actions, but to some generic
> watching, which would be the same in multiple PIM processes.
>

I'm not sure who is watching for emails. It would be nice to know. The
simplest way would be to add debug statements in the storage service
(resourcewatchermanager.cpp). I just looked at the code, and it can easily
be optimized quite a bit as well.


> > > I think using dbus is fine for notifications, but still, using it
> > > efficiently might be better :)
> > >
> > > I see nepomukservices using 100% CPU most of the time since I last
> updated
> > > kdepim stuff,
> > > and gdb says it's mostly busy preparing dbus messages to send out.
> >
> > Hmm. That's not good.
>
> That's mostly due to a bug in qdbusintegrator.cpp though (no cleanup of old
> dbus objects in the tree, cf my mail to Thiago and you).
>

Lets see if something can be done in time for 4.10


> > But I don't  think it would be simple to change the number of signals
> being
> > emitted. One thing that I have been thinking about it the use of
> > ResourceWatcher inside the Resource class. Though it keeps the cache up
> to
> > date, is it really worth it? Maybe we shouldn't do that add a
> > Resource::reload() function which reloads its contents. The client can
> > control that manually, if they really require updates.
>
> The client wouldn't know when to call reload().


They could always create their own ResourceWatcher!


> However, if there are use
> cases where updates are unlikely or unimportant, maybe a
> Resource::setAutoUpdating(false) would be good to have.
>

This seems like a better approach. But that would be adding a new API and
I'm not sure if I should be asking for another exception from the Release
Team. I'm sure they are quite tired of me. Though most of the applications
don't really need live updates.

Not sure what to do.


> But I really wonder who could possibly be watching my very old email; this
> must be some generic "watch all emails" thing? I guess I should add debug
> output in ResourceWatcher, but 1) it's used very very often in kmail, and
> 2)
> nepomuk resource URIs don't tell me much.
>
>
Right. Then it is definitely coming from the Resource class.

Try commenting out the details of the
Nepomuk2::ResourceManagerPrivate::addToWatcher function.


> > I think I should run the storage service in callgrind and see what is
> > actually going on. I've only done that for the file indexer so far, and
> > that was without any watchers.
>
> Well, profiling at the low-level is kind of what I just did.
>
> What would be more useful IMHO would be a higher-level picture (who is
> watching what, and for what purpose), to detect "stupid behavior" at the
> semantic (no pun intended) level.
>

Yes. That would be nice.

We need to do a rewrite of the entire NepomukShell and make it actually
return useful information like this. The storage service does know about
all of this "higer-level" stuff.


> David, who just realized his dbus service IDs are now up to :1.114565 --
> something is registering/deregistering very very often to dbus... this
> didn't
> use to be the case, hmm. Maybe threads coming and going, with their
> per-thread
> dbus-connection...
>
> --
> David Faure, faure at kde.org, http://www.davidfaure.fr
> Working on KDE, in particular KDE Frameworks 5
>
>


-- 
Vishesh Handa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/nepomuk/attachments/20130113/9a0a5d9d/attachment.html>