Fwd: OutputModel performance

[Milian Wolff]

On Thursday 07 February 2013 23:09:59 Andreas Pakulat wrote:
> Hi,
> 
> On Thu, Feb 7, 2013 at 6:37 PM, Milian Wolff <mail at milianw.de> wrote:
> >> > I can see that I accidentally sent this to Andreas only - sorry about
> >> > that.
> >> > 
> >> > A test like that already exists in outputmodeltest. It benchmarks the
> >> > different output filtering strategies.
> >> > 
> >> > Here is a sample from a test I just ran:
> >> > QDEBUG : KDevelop::OutputModelTest::benchmarkAddlinesNofilter() average
> >> > UI
> >> > lockup in ms:  0.263682
> >> > QDEBUG : KDevelop::OutputModelTest::benchmarkAddlinesCompilerfilter()
> >> > average UI lockup in ms:  8.64179
> >> > QDEBUG :
> >> > KDevelop::OutputModelTest::benchmarkAddlinesScriptErrorfilter()
> >> > average UI lockup in ms:  1.36318
> >> > QDEBUG :
> >> > KDevelop::OutputModelTest::benchmarkAddlinesStaticAnalysisfilter()
> >> > average UI lockup in ms:  0.228856
> >> > 
> >> > Maybe if we got a sample of the compileoutput that seems to cause
> >> > trouble,
> >> > rewrote the test a little to feed each line to the matching code and do
> >> > timing on matches. If one or more of the lines sticks out timing wise,
> >> > it
> >> > should be fairly simple to find the regex that performs badly on the
> >> > line(s).
> >> 
> >> Here's some sample output to play with: http://pastebin.com/FPEcUWP4
> >> (4400 lines, flushed in ~15 seconds). If this is done inside KDevelop, I
> >> get a rather sluggish UI with KDevelop keeping 1 core busy at 100%.
> >> 
> >> I can do some more profiling next week probably (busy writing exams atm).
> >> Then I can also have a more detailled look at the regexps used.
> > 
> > What should be done here:
> > 
> > a) Take a look at the RegExp's and try to optimize them (i.e. by making
> > them more explicit, i.e. adding ^,$ stuff or similar).
> 
> Many already have at least a ^ already. For the few ones that don't,
> some are make/cmake so should be easy to check wether a ^ makes sense
> (i.e. does not break the matching). But does anybody have fortran
> available? Since most/all of those don't have ^ or $. Also as you said
> (?:) can probably be added in more places.
> 
> > b) create an active worker object that is moved to a thread, then let that
> > do the classification. I.e.:
> > 
> > output line (QString) -> active object, run regexps -> (Categorized Data)
> > -> OutputModel
> 
> I'd rather have a worker-object running in a QThread-with-event-loop,
> this has a slot to give it lines and emits a signal with batches of
> parsed-data (i.e. 50 or so).

Hehe I should be more explicit. This is exactly what I had in mind.

> Then every time lines are added to the
> model, a signal is emitted which connects via queued-connection to the
> worker, that one adds the lines to its internal queue and processes
> lines. Once it has a batch of 50 done, it emits them and the
> outputmodel gets that through a queued-connection to add it to the
> model data and notify the view. That should be safe enough to not
> introduce many regressions and should solve the ui-lockup.

Yeah. Also note that we already do batch processing just minus the thread.

> > c) also ensure the regexps are only compiled once - but I think that is
> > already the case, no?
> 
> Yes, they're static objects.

OK, cool.

> > d) wait for Qt5 - the regexp stuff there is much much better
> 
> Or use  pcre directly today.

Considering that we can relatively easily work around this as outlined above 
I'd rather not introduce a dependency on pcre here just for that. Esp. not 
considering that we'll have to rewrite that stuff for Qt5 anyways *and* get 
proper pcre there then.

Cheers
-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20130207/b75a7650/attachment.sig>