kdevelop 4 parsing on opensuse 11.2 is extremely slow for linux kerenl project

[Milian Wolff]

On Saturday 03 April 2010 18:40:58 David Nolden wrote:
> I'm trying to answer now to all the mails together, including your
> personal one, Milian.
> 
> PERFORMANCE:
> In general, the include-path resolver _already_ caches the result
> per-directory, and usually that works! The big slowdown you're seeing
> on the kernel is probably a plain and simple bug.

Caching was broken since operator== in ModificationRevisionSet was broken ;-) 
How could that have slipped us so long?

> I think the main problem is that only _successful_ include-path
> resolver runs are cached. When the runs are never considered
> successful, then they are never cached, and we run it over and over
> again.
> 
> In the case of the kernel, the runs are probably _never_ considered
> succesful, because there simply is no include-paths.

I've fixed that for the kernel at least: There make failed even when it 
returned paths. Now I ignore the return value more or less.

> So there would be 2 things to do here:
> - Make sure we recognize a run as successful even if the result is
> empty (kernel case)

problematic, since make fails for me even for the .ko cases...

> - Make sure we don't unsuccessfully call "make" over and over again.
> The problem here is, that it's hard to pick the 'correct' make call.
> For example, there may be some files in a directory that simply don't
> have a target. Calling "make" for those will fail, and we don't want
> to cache that for all the remaining files in the directory.

I'm note sure how to do that properly, and imo the current code is error-prone 
enough, I don't want to add more :-/

> It might make sense to aggregate all the make calls for one file
> within one call, I don't exactly remember why the exit-status was
> needed for each of them, I think it was so that we can identify the
> one that was successful.

Done that, also for relative & absolute calls. Should work for make, but I 
don't have a clue about unsermake...

> Now that we're doing per-directory caching, it might even make sense
> to completely ditch the selection of a specific target, and simply
> call "make" without a target specification, and give it
> "--assume-changed" switches for all the files in the directory.

Yeah that could be a good idea, dunno.

> MODIFICATION-TIME ISSUES:
> Currently the include-path resolver changes the modification-times of
> the files it tries to resolve include-paths for, by doing a "touch",
> so that the gcc command line will actually get printed. When it does
> that, then it remembers the old modification-time, and resets it once
> the include-path resolver run has completed, and if the file hasn't
> been changed in between. The problem is that there simply is no way to
> do this 100% without issues, because there is always a gap between the
> check "should we reset the modification-time?" and the actual reset.
> 
> At the same time the "--assume-new" option of make is used as well,
> but from what I remember there have been issues with this because of
> the way the file is specified (absolute/relative etc.), and it didn't
> work together with additional scripts like unsermake (though this
> might be superseeded by now).

I'd like to see a reproducable case for that. I really want to get rid of that 
blind touching, I really have the gut feeling that it's the cause for these 
random issues we get in regard to timestamp changes on files etc. pp.

> However, if the caching works correctly, this should be a non-issue,
> as the modification happens only once for one file in each directory.
> Still, this should be re-investigated. Actually in the beginning I was
> planning to use the modification-time changing only if there is an
> underlying script like unsermake, but something like that is hard to
> decide.

If you give me more infos, I can investigate it deeper, but without more I'm 
lost....

> USEFULNES:
> The include-path resolver is useful, and should stay there. It makes
> the C++ support "just-work" for cmake-, automake, unsermake, and
> simple custom-makefile projects, and I think even qmake projects,
> which altogether should cover 90% of the common build-systems under
> linux. The only project-manager we can half-way reliable get
> include-paths by ourselves right now is cmake, but given that cmake is
> permanently evolving, I still wouldn't want to rely 100% on custom
> parsing of it.
> 
> The issues we have right now can be solved through better caching of
> the results. In future we might consider adding some switches for the
> include-path resolver: Disabled, per-project, per-directory, per-file.

I agree that it's kinda nice-to-have for some cases, but as far as I can see 
it has _no_ usecase for cmake at all, except in-source-builds (which I hope 
are nearly non-existent...) I mean on the CLI (duchainify) we have no way to 
specify the build-dir, hence it will never find the Make files.

In the future I'd still prefer using Apaku's plugin for include paths and only 
keep this as last-case-resort...

> TESTING:
> Unfortunately there is no unit-tests, but there is an
> "includepathresolver" tool that can be compiled through a cmake
> switch, and that can then be used to do some testing.

That tool totally blows :D

I don't get how to use it properly... It should probably rewritten to 

params: 1. file [2. source-directory 3. build-directory]

and than turn file to an abs path and call the resolvePath on that (i.e. 
workingDir == path of file). Or why should one use the other case? Esp. given 
that the behavior above is also used in the IncludePathComputer...

And real unit tests would really help, but I fear I'm not the right person for 
that. I don't know what most of this code is supposed to do ;-) I hope I 
didn't break anything with my recent commits. It would be really cool if you 
could review them.
-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20100405/835c97df/attachment.sig>