D12746: KDevelop: alternative monolithic storage options for TopDUContexts (WIP/PoC)

[Milian Wolff]

mwolff added a comment.


  so what's your plan with this now? Do you actually want to move this forward? Then we need to find a good solution that works everywhere. The numbers aren't that useful on their own, as there the current implementation which is trivial and has no external dependencies is apparently still the best by a margin...
  
  I'm really quite unsure if we want to go down this route. If we'd decide to do that, then it would need to be done also in the other places, most notably to replace our actual performance sensitive databases. The current topconteext data loading is pretty fast. What's slow, or at least used to be slow, is hammering the big databases we have (i.e. AbstractItemRepository). *These* are the slow ones, since they are actually used a lot. Still, the current implementation was much faster than lmbd the last time I tried. The reason is that we can access and insert quickly, and lookup is O(1) since it's an on-disk hash map. But actually, I would very much appreciate getting *rid* of that complicated code and replacing it by something else. The reason is that we still have lots of crash reports in that code that are quasi impossible to debug/reproduce. Also, if we'd use an established data base implementation, we could potentially make access cross-process safe, which I would really like to see in the long term.
  
  So before I do any review, we first need to talk about the above.

INLINE COMMENTS

> test_topcontextstore.cpp:76
> +// PASS   : TestTopDUContextStore::benchTopContextStore()
> +// QINFO  : TestTopDUContextStore::benchTopContextStoreRead() Reading 1500 keys: 0.13 seconds
> +// PASS   : TestTopDUContextStore::benchTopContextStoreRead()

you are missing the 6k results here

> test_topcontextstore.cpp:145
> +// PASS   : TestTopDUContextStore::benchTopContextStoreRead()
> +// QINFO  : TestTopDUContextStore::benchTopContextStoreRemove() Removing 1500 keys: 7.234 seconds
> +// PASS   : TestTopDUContextStore::benchTopContextStoreRemove()

that's pretty bad in comparison

note that we currently just rewrite the complete file, which you can't do when you use one big data base

> test_topcontextstore.cpp:355
> +// this is actual TopDUContext data from KDevelop itself
> +unsigned char realDUData[] = {64, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 127, 255, 255, 255, 127, 
> +64, 64, 140, 0, 2, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 127, 0, 0, 0, 1, 

O_o make it a separate data file and read it from a resource instead of embedding it manually

REPOSITORY
  R32 KDevelop

REVISION DETAIL
  https://phabricator.kde.org/D12746

To: rjvbb, brauch
Cc: mwolff, kdevelop-devel, antismap, iodelay, vbspam, njensen, geetamc, Pilzschaf, akshaydeo, surgenight, arrowdodger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20180507/ad3901cc/attachment.html>