[RFC] [kservice] KPluginMetadata indexing

[Milian Wolff]

On Thursday 06 November 2014 17:06:38 Mark Gaiser wrote:
> On Thu, Nov 6, 2014 at 4:43 PM, Sebastian Kügler <sebas at kde.org> wrote:
> > On Thursday, November 06, 2014 10:09:51 Mark Gaiser wrote:
> >> I'm curious about one thing. Have you done some profiling on the
> >> current KPluginMetaData to see where the actual hot spot is?
> >> In case you don't know how to do that, here are some tips:
> >> 1. Recompile Qt with debug symbols (not debug mode, just with the debug
> >> symbols) 2. Run a benchmark application via valgrind like so: valgrind
> >> --tool=callgrind <your_benchmark_app>
> >> 3. Open the output file of the line above in KCacheGrind and hunt for
> >> those pesky hot spots.
> >> 
> >> Perhaps there is nothing to optimize and then having an index (and the
> >> cost of maintaining it) is worth it, but it would be best to first
> >> determine if the current code path can be optimized.
> > 
> > I've focused on reducing the I/O, since that's where we spend by far most
> > of the time, somewhere beyond 90% of the whole time it takes to run the
> > query (and in worse cases with cold caches and on rotational media, even
> > more).
> > 
> > The next bottleneck would be the deserialization of json data, which is
> > for
> > the binary format we use for storage (and I think which is also used in
> > QPluginLoader to read the plugins json metadata). As fas as I can see,
> > that is also pretty much entirely I/O bound.
> > 
> > There may be something to be gained in the conversion from the
> > KPluginMetaData list to the KPluginInfo::List by making the query runner
> > 'understand' KPluginMetaData (it knows KService and KPluginInfo
> > currently) but that's definitely not a bottleneck currently.
> > 
> > In the whole picture, KPluginMetaData is not a concern right now, reducing
> > the I/O is what we need to do first.
> > 
> > Cheers,
> 
> Ah right, i think i missed the I/O reducing goal in your initial post.
> 
> I haven't checked your draft implementation in detail, but what you
> might want to do is:
> - Monitor all plugin folders for changes
> - If a change in any of the folders is detected: rebuild cache.
> 
> That should keep the cache updated in a fairly easy way and remove the
> need to have a tool at all.
> You could do it more fine grained and only update the cache for the
> plugin that changed. It will be a bit more tricky to implement.
> 
> Just an idea, but i hope it helps.

Most linux distros have a strict limit of file watches. Baloo, KDevelop and 
other projects already battle for that resource. Adding one more watcher won't 
simplify that situation. Generally, there will be occasions where adding a 
watcher failed. You cannot rely on that feature to work reliably.

bye
-- 
Milian Wolff
mail at milianw.de
http://milianw.de