kdevelop 4 parsing on opensuse 11.2 is extremely slow for linux kerenl project

[Milian Wolff]

On Friday 02 April 2010 14:50:30 Christoph Bartoschek wrote:
> Am Freitag 02 April 2010 schrieb Christoph Bartoschek:
> > I've also tried to import the linux kernel tree. However I do not see
> > that parsing is slow. I would say that the problem lies in the
> > IncludePathResolver. My observations are:
> > 
> > 1. During importing kdevelop only takes 10-25% CPU time.
> > 2. The IncludePathResolver changes the timestamps of the files all the
> > 
> >  time and it seems as if a FileSystemWatcher is fired because of the
> >  changes within kdevelop. (Why is the resovler not using the option
> >  --assume-new provided by make? At least if the make is GNU make)
> 
> I've just seen in the code that there is code that is able to use the --
> assume-new option. However the timestamps are still always updated. Why is
> this not enabled?
> 
> I've made the following experiment. I've disabled the IncludePathResolver:
> it always returns false.
> 
> Now the whole linux kernel is initialy parsed in 15 minutes on my machine.
> I've tried to load it with the Resolver enabled but gave up after 2 hours
> and 5% loaded.

Hihihihi I had a quick look at it and you where spot on, the 
IncludePathResolver is really slow. Why? Because it calls this make command 
_for every single file_:

time make -k --no-print-directory -W '/home/milian/projects/linux-2.6.34-
rc3/include/linux/i2o-dev.h' -n include/linux/i2o-dev.ko

(paths get changed of course).

When I run this make cmd alone:
real    0m1.482s
user    0m0.597s
sys     0m0.420s

But look at this:  http://pastebin.com/bN2E8vxa
It's actually getting called more than once ;-) And that's what makes things 
so horribly slow for custom make files. It's never occurred to us developers 
since all of us are only using cmake where this bottleneck doesn't occur. Oh 
and since it's executing an external app and waits for it to finish, it doesn't 
show up in callgrind _at all_ :-/

I'll now try to implement this:

///@todo Resolve include-paths either once per directory, or once per target. 
Do not iterate over the duchain.

And see how much of a differnce that makes. But I'm confident that this can 
easily be an order of magnitude.

> In my experiment I used 4 parser threads but I see only 144% CPU time on my
> 4-core-machine. This means that kdevelop still wastes lots of time waiting.
> I do not know for what it waits: mutexes or I/O.  Unfortunately I do not
> know a good profiler that is able to analyze waiting time. Callgrind is
> definitively the wrong one.
> 
> Christoph

-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20100403/0f1b2069/attachment.sig>