Very slow background parsing

Dmitry Risenberg dmitry.risenberg at gmail.com
Thu Dec 30 18:04:30 UTC 2010


Investigated this case a little more:

1) The code that causes reparsing is preprocessjob.cpp:609 - if an
include file is found that was missing on previous parsing run, then
recursive reparsing is called. This is OK with one file, but if an
include path was found for a project that was absent before, this will
cause a recursive reparsing for all files affected, which may be quite
many. This is why the parsing process became slow. It looks like an
error to me - why reparse the system includes for each file? They
should be added to a set of parsed files, and when a new file is to be
reparsed, do nothing if it already in the set.

Where do the missing paths come from? I tracked them down to
CPPParseJob::includePaths, where I got some questions.

2) Looks like the paths are computed twice, first with a call to
findIncludePathsForJob in cppparsejob.cpp:240, then with
m_includePathsComputed->computeBackground() in cppparsejob.cpp:251.
Why is this necessary? And why invokeMethod and waiting on mutexes
here, if we want the paths to be found synchronously?

3) Sometimes I get a log message

kdevelop(9882)/kdevelop (cpp support)
IncludePathComputer::computeForeground: Did not find a build-manager
for KUrl("file:///home/drisenberg/Development/KDevelop/kdevelop/languages/cpp/preprocessjob.cpp")

and in this case, the set of project-specific include paths returned
by IncludePathComputer::computeForeground is empty. Other times the
paths are computed correctly, so I assume there is a race condition
here, but don't know between what code - hope someone with a better
knowledge of design will clarify that =)

2010/12/29 Dmitry Risenberg <dmitry.risenberg at gmail.com>:
> Hi all!
>
> Recently the background parser has become very slow on my machine - 30
> minutes after startup it is only about 40% finished. I wasn't able to
> find the reason for that - rolling back to an older repository
> snapshot did not fix it, and no relevent updates to the system itself
> were installed. Besides, this affected only one of my two PCs.
> This also doesn't seem to be the bug with zlib version - I have
> "1.2.3" as version.
>
> The only clue about such behaviour that I did find are entries in the
> debug output like the following one:
>
> kdevelop(23360)/kdevelop (cpp support) PreprocessJob::sourceNeeded:
> PreprocessJob "/home/drisenberg/Development/KDevelop/kdevelop/build/languages/cpp/ui_custom_include_paths.h"
> : need to update KUrl("file:///usr/include/qt4/QtGui/QHeaderView")
> kdevelop(23360)/kdevelop (cpp support) PreprocessJob::run:
> PreprocessJob: preprocessing "/usr/include/qt4/QtGui/QHeaderView"
> kdevelop(23360)/kdevelop (cpp support) CPPInternalParseJob::run: ===--
> PARSING --===>  "/usr/include/KDE/KGlobal"
> kdevelop(23360)/kdevelop (cpp support) CPPInternalParseJob::run:
> building duchain for "/usr/include/KDE/KGlobal"
> kdevelop(23360)/kdevelop (cpp support) ContextBuilder::buildContexts:
> ContextBuilder::buildContexts: compiling
> kdevelop(23360)/kdevelop (cpp support) ContextBuilder::buildContexts:
> built top-level context with 0 declarations and 9 included files
> kdevelop(23360)/kdevelop (cpp support)
> ContextBuilder::buildProxyContextFromContent:
> ContextBuilder::buildProxyContextFromContent: compiling
> kdevelop(23360)/kdevelop (cpp support) CPPInternalParseJob::run: ===--
> Parsing finished --===> "/usr/include/KDE/KGlobal"
> kdevelop(23360)/kdevplatform (language)
> KDevelop::DUChainPrivate::loadChain: loading top-context 2293
> kdevelop(23360)/kdevplatform (language)
> KDevelop::DUChainPrivate::loadChain: loading top-context 1597
> kdevelop(23360)/kdevelop (cpp support) PreprocessJob::sourceNeeded:
> PreprocessJob "/usr/include/qt4/QtGui/QHeaderView" : need to update
> KUrl("file:///usr/include/qt4/QtGui/qheaderview.h")
>
> There are multiple entries with such pattern: a Qt-generated header
> file calls reparsing of a Qt header, which in turn reparses its own
> includes, and so on. This is strange, because
> ui_custom_include_paths.h doesn't define any macros before the
> includes, except for the guard, so there's no need for an update.
>
> I also tried running duchainify on kdevelop project directory with
> --force-update-recursive, which gave much better time - 11 minutes. In
> the log there were the following entries:
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude: FAILED
> to find include-file "QtGui/QHeaderView" in paths:
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/include/c++/4.4")
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/include/c++/4.4/x86_64-linux-gnu")
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/include/c++/4.4/backward")
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/local/include")
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/lib/gcc/x86_64-linux-gnu/4.4.5/include")
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/lib/gcc/x86_64-linux-gnu/4.4.5/include-fixed")
> duchainify(18660)/kdevelop (cpp support) CppUtils::findInclude:
> KUrl("file:///usr/include")
> duchainify(18660)/kdevelop (cpp support) PreprocessJob::sourceNeeded:
> PreprocessJob "../../kdevelop/build/languages/cpp/ui_custom_include_paths.h"
> : include not found: "QtGui/QHeaderView"
>
> So may be duchainify was showing a better time because it just failed
> to do the full re-parsing.
>
> I'm not attaching the full parsing logs here, because they are quite
> big (2.5 Mb zipped), but can attach them if necessary.
>
> --
> Dmitry Risenberg
>



-- 
Dmitry Risenberg




More information about the KDevelop-devel mailing list