[Kde-pim] Nepomukfeeder Caching

[Christian Mollekopf]

Hey,

I found some time & motivation to work on the feeders again. One of the
things 
I've been working on is a cache, to implement Visheshs suggestion.
It's a generic cache, which caches Nepomuk::SimpleResources by type. I
didn't 
feel like implementing little caches all over the place, so this was
more fun 
to develop and does the job nicer as well. (Since it's only tied to
Nepomuk it 
could also become part of the nepomuk-core framwork in the future.)

The cache relies on a couple of assumptions:

* The hashing function simply xor's the conjunction of the hashes of uri
and 
value of each resource-property together to calculate the hash of the
full 
resource.  I don't know the math behind that but simply copied it from 
nepomuk-core, input would be appreciated:
propertycache.cpp:50

* I assume that if a resource refers to a subresource using a temporary
"_:" 
uri, that resource is always available in the same SimpleResourceGraph.
I'm 
fairly certain on that one.

So the cache simply stores hash-resourceUri pairs, and removes resources
which 
have already been cached from the final SimpleResourceGraph and replaces
the 
references to it with the absolute resourceUri.

First tests with mail look promising, I'm getting performance
improvements 
from 100-200% with a cache hit ratio of 0.5-0.8 (of the cached types
only), 
where mailinglists are on the higher end and normal folders on the lower
end.
I expected the continuous hashing of every resource would show up as
well, but 
it's not noticeable on my machine, also the additional memory
consumption is 
marginal as it looks like 100 cached items are already enough.

I'm currently caching:
* nco#EmailAddress
* nco#Contact
* nao#FreeDesktopIcon
* nmo#MessageHeader

The working but WIP code is currently here:
http://commits.kde.org/clones/kdepim-
runtime/cmollekopf/pimRuntimeClone/016ea4dd527f34a8bc6b08c9f8ca9eef2c412f15

All in all a rewarding exercise 

I'd like that in 4.10 together with some other improvements, I hope
that's 
possible

I did quite some brainstorming on other problematic areas of the
feeders, I 
hope to do some more improvements in that area.
Especially I hope to not rely too much on the ChangeRecorder, so we can
turn 
it off if Nepomuk or the feeders are turned off, to keep the log from
growing 
endlessly.

Cheers,
Christian
_______________________________________________
KDE PIM mailing list kde-pim at kde.org
https://mail.kde.org/mailman/listinfo/kde-pim
KDE PIM home page at http://pim.kde.org/