Very slow background parsing

[David Nolden]

2010/12/30 Dmitry Risenberg <dmitry.risenberg at gmail.com>:
> Investigated this case a little more:
>
> 1) The code that causes reparsing is preprocessjob.cpp:609 - if an
> include file is found that was missing on previous parsing run, then
> recursive reparsing is called. This is OK with one file, but if an
> include path was found for a project that was absent before, this will
> cause a recursive reparsing for all files affected, which may be quite
> many. This is why the parsing process became slow. It looks like an
> error to me - why reparse the system includes for each file? They
> should be added to a set of parsed files, and when a new file is to be
> reparsed, do nothing if it already in the set.

At least in C++, this is unfortunately required. The comment there
basically says it all. Following parsed files may contain references
to items that are visible only with the added include file. It's
unprobable that system files reference your project files, but in
theory they could (you could override a macro, for example).

Unfortunately this all is a big mess when parsing C++. In the
beginning I've tried to solve this properly using environment
management, but a too high effort was required to track all the
environment dependencies properly, so I've disabled it before the 4.0
release, and we're back to a somewhat "stupid" but simple system that
doesn't really track the dependencies. However, at least from what I
read around, most other IDEs do the same.

> Where do the missing paths come from? I tracked them down to
> CPPParseJob::includePaths, where I got some questions.
>
> 2) Looks like the paths are computed twice, first with a call to
> findIncludePathsForJob in cppparsejob.cpp:240, then with
> m_includePathsComputed->computeBackground() in cppparsejob.cpp:251.
> Why is this necessary? And why invokeMethod and waiting on mutexes
> here, if we want the paths to be found synchronously?

The whole code is called from within a background thread. However, the
IncludePathResolver::computeForeground() simply asks the
project-manager, and that one is only accessible from within the
foreground thread. Thus the invokeMethod, mutexes, etc. These days,
this whole ugly construct could be replaced by a simple call to the
foreground lock.

Later we call computeBackground() within the background. That one
eventually calls "make" to parse the "gcc" command line and thereby
retrieve the include paths. This is slow, so it's important to do it
within the background, so we don't block the UI.

> 3) Sometimes I get a log message
>
> kdevelop(9882)/kdevelop (cpp support)
> IncludePathComputer::computeForeground: Did not find a build-manager
> for KUrl("file:///home/drisenberg/Development/KDevelop/kdevelop/languages/cpp/preprocessjob.cpp")
>
> and in this case, the set of project-specific include paths returned
> by IncludePathComputer::computeForeground is empty. Other times the
> paths are computed correctly, so I assume there is a race condition
> here, but don't know between what code - hope someone with a better
> knowledge of design will clarify that =)

That's a bug in the project-manager i suppose (or maybe the project
hasn't finished loading yet?)

Unfortunately, it's hard to come up with a real solution to this
problem, because the whole preprocessor/environment stuff is very
complicated.

Greetings, David