Fwd: OutputModel performance

[Andreas Pakulat]

Hi,

On Thu, Feb 7, 2013 at 6:37 PM, Milian Wolff <mail at milianw.de> wrote:
>> > I can see that I accidentally sent this to Andreas only - sorry about
>> > that.
>> >
>> > A test like that already exists in outputmodeltest. It benchmarks the
>> > different output filtering strategies.
>> >
>> > Here is a sample from a test I just ran:
>> > QDEBUG : KDevelop::OutputModelTest::benchmarkAddlinesNofilter() average UI
>> > lockup in ms:  0.263682
>> > QDEBUG : KDevelop::OutputModelTest::benchmarkAddlinesCompilerfilter()
>> > average UI lockup in ms:  8.64179
>> > QDEBUG : KDevelop::OutputModelTest::benchmarkAddlinesScriptErrorfilter()
>> > average UI lockup in ms:  1.36318
>> > QDEBUG :
>> > KDevelop::OutputModelTest::benchmarkAddlinesStaticAnalysisfilter()
>> > average UI lockup in ms:  0.228856
>> >
>> > Maybe if we got a sample of the compileoutput that seems to cause trouble,
>> > rewrote the test a little to feed each line to the matching code and do
>> > timing on matches. If one or more of the lines sticks out timing wise, it
>> > should be fairly simple to find the regex that performs badly on the
>> > line(s).
>>
>> Here's some sample output to play with: http://pastebin.com/FPEcUWP4
>> (4400 lines, flushed in ~15 seconds). If this is done inside KDevelop, I get
>> a rather sluggish UI with KDevelop keeping 1 core busy at 100%.
>>
>> I can do some more profiling next week probably (busy writing exams atm).
>> Then I can also have a more detailled look at the regexps used.
>
> What should be done here:
>
> a) Take a look at the RegExp's and try to optimize them (i.e. by making them
> more explicit, i.e. adding ^,$ stuff or similar).

Many already have at least a ^ already. For the few ones that don't,
some are make/cmake so should be easy to check wether a ^ makes sense
(i.e. does not break the matching). But does anybody have fortran
available? Since most/all of those don't have ^ or $. Also as you said
(?:) can probably be added in more places.

> b) create an active worker object that is moved to a thread, then let that do
> the classification. I.e.:
>
> output line (QString) -> active object, run regexps -> (Categorized Data) ->
> OutputModel

I'd rather have a worker-object running in a QThread-with-event-loop,
this has a slot to give it lines and emits a signal with batches of
parsed-data (i.e. 50 or so). Then every time lines are added to the
model, a signal is emitted which connects via queued-connection to the
worker, that one adds the lines to its internal queue and processes
lines. Once it has a batch of 50 done, it emits them and the
outputmodel gets that through a queued-connection to add it to the
model data and notify the view. That should be safe enough to not
introduce many regressions and should solve the ui-lockup.

> c) also ensure the regexps are only compiled once - but I think that is
> already the case, no?

Yes, they're static objects.

> d) wait for Qt5 - the regexp stuff there is much much better

Or use  pcre directly today.

Andreas