[Nepomuk] Redundant DBus communication

[David Faure]

On Sunday 13 January 2013 17:12:15 Vishesh Handa wrote:
> You're right. We can possibly remove the added and removed signals and
> infer that data from the changed signal.

Won't this require expensive lookups on the receiver side?
(If not, then I agree.)

The alternative would be to emit added without emitting changed right away -- 
but I don't know the current design (= what happens with these signals).

> They'll each have their own. This has mostly been done cause it is really
> hard to implement it otherwise.
> 
> On the server side we have 3 hash tables which maps the following keys  -
> * Resources
> * Properties
> * Types
> 
> to each ResourceWatcher.

Yes, and I wonder if it happens often that the exact same set of keys is 
requested by the same code in multiple applications.
In kdepim, I suspect this might happen? Or maybe I don't really understand the 
context. It's worth checking, at least.

> Each ResourceWatcher can watch a number of different combinations of all of
> these (They conditions are ANDed). Therefore it is just a lot simpler for
> each for each ResourceWatcher to listen to their own dbus path, instead of
> creating special dbus paths for the intersection of resource/properties and
> types.

Right. I'm not suggesting intersections, but equality tests.

If the same very very old email of mine is watched by more than one process, 
I'm pretty sure it's not related to my own actions, but to some generic 
watching, which would be the same in multiple PIM processes.

> > I think using dbus is fine for notifications, but still, using it
> > efficiently might be better :)
> > 
> > I see nepomukservices using 100% CPU most of the time since I last updated
> > kdepim stuff,
> > and gdb says it's mostly busy preparing dbus messages to send out.
> 
> Hmm. That's not good.

That's mostly due to a bug in qdbusintegrator.cpp though (no cleanup of old 
dbus objects in the tree, cf my mail to Thiago and you).

> But I don't  think it would be simple to change the number of signals being
> emitted. One thing that I have been thinking about it the use of
> ResourceWatcher inside the Resource class. Though it keeps the cache up to
> date, is it really worth it? Maybe we shouldn't do that add a
> Resource::reload() function which reloads its contents. The client can
> control that manually, if they really require updates.

The client wouldn't know when to call reload(). However, if there are use 
cases where updates are unlikely or unimportant, maybe a 
Resource::setAutoUpdating(false) would be good to have.

But I really wonder who could possibly be watching my very old email; this 
must be some generic "watch all emails" thing? I guess I should add debug 
output in ResourceWatcher, but 1) it's used very very often in kmail, and 2) 
nepomuk resource URIs don't tell me much.

> I think I should run the storage service in callgrind and see what is
> actually going on. I've only done that for the file indexer so far, and
> that was without any watchers.

Well, profiling at the low-level is kind of what I just did.

What would be more useful IMHO would be a higher-level picture (who is 
watching what, and for what purpose), to detect "stupid behavior" at the 
semantic (no pun intended) level.


David, who just realized his dbus service IDs are now up to :1.114565 -- 
something is registering/deregistering very very often to dbus... this didn't 
use to be the case, hmm. Maybe threads coming and going, with their per-thread 
dbus-connection...

-- 
David Faure, faure at kde.org, http://www.davidfaure.fr
Working on KDE, in particular KDE Frameworks 5