plugins,caches, startup time

[Sebastian Kügler]

Hi,

On Tuesday, September 27, 2016 4:52:22 PM UTC Marco Martin wrote:
> as there is being some work in plugin caching right now, to see if /what we
> can save at startup time would be nice if different people on different
> systems do some tests.
> on his system, by indexing plugins Sebas saves about 2-300 milliseconds. How
> did you test that, so that we see wether we can reproduce it on different
> systems, (may well be that on some kinds of hardware it's worth it, not on
> some other)

My measuring was done as follows:
- I instrumented Plasma::PluginLoader and recorded all calls to 
KPluginLoader::findPackage(...)
- I put these queries into an autotest, and instrumented that with timers that 
measure the time spent in KPluginLoader::findPlugins(...), with and without 
cache.

So my tests are a little more synthetic. My code is in kcoreaddons[sebas/
pluginindexer2], the index is generated with 

$ kpluginindexer -u 

Results on my fast system (core i7 + ssd) is that for Plasma startup, the 
indexed code runs about 200ms faster (out of a total ~700). That's not a lot 
of gain, IMO, and I'm unsure if it justifies the complexity added by 
maintaining an index.

I haven't done any measurements on cold caches.

> I did a similar test on kpackage plugin indexes, that's how all plasmoids
> get loaded, so it's quite a lot of them.

Nice, I suppose KPackages are a little different to plugins, since there's 
more overhead reading, the package metadata is stored as .desktop file, which 
means more conversion costs compared to the binary json that's in the plugin 
binaries. Aleix has recently been looking into converting it to json, but that 
would only save the .desktop to json part, binary json is still much quicker 
to read (the format is pretty close to Qt's internal representation in memory, 
deserialization is almost entirely I/O-bound.

> to measure the plasma startup i use
> https://phabricator.kde.org/D2835
> (probably not worth to push it, just easy to apply the patch with arc)
> the elapsed time there is taken when the panel appears, because it's where
> we are more reasonably certain the startup is done (that would be,
> where/when the last panel appears)
> 
> to generate the plugin index, use:
> kpackagetool5 --generate-index -g -t Plasma/Applet

I suppose we can save more by indexing all packages?
Also, in the current code, we discard the cache after each query, so we get 
the nasty i/o time and time again, for each kpackage query. One thing that 
would again boost the performance is if we cached the binary json files once 
we've read them, even if we discard the cache after some time. There are a lot 
of consecutive queries, and by far the biggest cost in this codepath is now 
the reading of the index. I think it would be quite interesting to do coarsely 
by keeping a QHash<filepath, jsondocument> with the data we've read. (To save 
some runtime memory, we could clear this say 1 minute after its last use.)

One thing that the code in kpackage doesn't do yet is to verify that the index 
is up to date, that isn't hard but needs implementing. I've done it in my 
kpluginindexer2 branch by comparing mtime of the plugin directory and the 
cache file. Slows things down a little bit of course, but it actually makes 
pretty sure we're not operating from an outdated cache file, see 
KPluginIndexer::isCacheUpToDate(...).

> on an ssd system i save around 5-600 milliseconds on a startup time around 5
> seconds (on a debug build of qt, which i note it matters a lot for startup
> time of qml stuff, on the same laptop a plasma session installed from neon
> packages takes less than half to start)
> on an old spinning metal workstation, still debug build of qt, i save around
> a second on top of a ~10 seconds startup time
> 
> it's not much, but a considerable amount, consistent around ~10%

It does all add up, but the overhead of generating the index and keeping it 
updates it also considerable, it needs changes trickling down into every 
package installing a plugin.
-- 
sebas

Sebastian Kügler    •    http://vizZzion.org    •    http://www.kde.org