GSoC 2010: Making KDevelop suitable for large C projects

[Milian Wolff]

On Monday 29 March 2010 19:46:06 Nitin Gupta wrote:
> Hi,
> 
> On 03/29/2010 06:03 PM, Milian Wolff wrote:
> > Nitin Gupta, 27.03.2010:
> >> For past few months, I have been working on a simple IDE (called KXref)
> >> based on Sublime UI framework, to allow working with large C based
> >> projects (Linux kernel in particular): http://code.google.com/p/kxref/
> >> 
> >> However, I soon realized that it will be much more useful and productive
> >> if all this can be developed as plugins for upcoming KDevelop4. My
> >> initial attempt to get it working for Linux kernel failed badly
> >> primarily due to lack of cscope, or better GNU global, integration
> >> (built-in background parser is too slow for such large projects). There
> >> are many other factors which make it unsuitable for kernel projects (as
> >> detailed below).
> >> 
> >> Unlike popular belief, an IDE for kernel projects (IMHO) is highly
> >> desirable. Trying to understand such huge and complex codebases is
> >> almost impossible otherwise. Plain vim+ctags is often not sufficient.
> >> 
> >> I'm planning to submit this as a GSoC proposal. I would be thankful for
> >> any feedback.
> > 
> > My personal opinion is: One can integrate each and every tool there is
> > for KDevelop, but I'd much prefer improving the performance of the build
> > in parser instead of using yet another tool for the job...
> > 
> > I found quite a few places to optimize over the weeks, and I bet there
> > are more.
> 
> Built-in parser seems to be slow beyond repair. Following is test summary:

*sigh*

>  * Project 1 (stripped down linux kernel tree)
>     - Size: ~18000 c, h files
>     - Time for backgroud parser to complete till 10%: 18 mins
>     - I closed KDevelop at this time. But at this rate, it would have taken
> 3 hrs!! - Time taken by GNU global for 100% completion (gtags): 9 seconds.
> - Restarted KDevelop. Now the parser starts from beginning! Expected it to
> start from 10%.

No of course it doesn't start from 10%. It starts from 0% since it has to 
check whether any file got changed in the meantime.

What it should do is being faster. And that it is.

>  * Project 2 (stripped down above tree even further)
>     - Size: ~600 c, h files
>     - Time taken by in-built parser (100% completion): 1 min 30 seconds
>     - Time taken by gtags: 0.6 seconds
>     - Verified that the in-built parser created database in ~/.kdevduchain
> however each time KDevelop is restarted, it seems to *recreate* the same
> database. It seems only dumping the cross reference db is currently
> supported and not really reusing it.

Wrong. KDevelop does use the cache.

Another thing: If you are doing profiling, please built KDevelop in 
RelWithDebInfo mode. Than use the duchainify utility to measure times.

To get actual helpful information other than "its slow, one cannot repair it" 
use Valgrind with Callgrind to produce profiles which we can use to make it 
actually faster.

> I have not yet started digging into code but I guess above stark difference
> in gtags vs in-built parser speed is due to much more contextual
> information gathered by the latter (like usage of a local variables
> withing functions etc), but now quite sure.

Most probably the amount of data we store. And the fact that there are still 
corners left to optimize. What I'd also be interested in is whether you 
cleaned gtags' tag files inbetween or not. If you did - how fast is it on a 
cached run?

> Considering above, I think it makes sense to implement GNU Global (gtags)
> plugin to handle large (especially C) projects. At the same time, use
> in-built parser to provide rich local context information -- like
> highlighting usage of a local variable within its scope -- for currently
> opened files only.

Feel free to do that. KDevelop is insanely modular and I think one can do 
about anything with plugins. If anything is missing in the API let us know.

> GNU Global claims to support C, C++, Yacc, Java and PHP4 so this plugin
> might be useful for large projects based on all these languages but I'm
> not really sure how good is gtags support for languages other than C.

As the maintainer of the PHP plugin I can safely say that the internal 
structure is "fast enough" even now. Remember that it's all done in the 
background and currently opened files get precedence. Hence you can work while 
kdevelop parses stuff in the background without much impact.

> Apart from faster parsing, essential components for supporting large C
> projects are: - Graph Viewer: Use information from gtags query to quickly
> generate call graphs - Check if existing graph viewer can be extended to
> use gtags or a new one needs to be developed.

If you want gtags, you need your own plugin. The good thing about the DUChain 
is that it's language agnostic. That also means that e.g. class browser, 
outline, quickopen, controlflowgraph, ... plugins only need to be written once 
to work for the multitude of languages we have even now (C++, PHP, Css, 
Erlang, Python, ...)

> Query Viewer: Again use
> gtags for queries and show results as generated by gtags - Current 'Code
> Browser' window does not show query history.

Code browser knows about history, see various shortcuts and the code browser 
toolbar. It's just the toolview not showing these.

> A tabbed query result window
> is needed to maintain query history (very important).
>  - Generic C template(s)
>  - Linux kernel specific template(s)
> 
> I think all above is doable within GSoC timelines (I will include project
> schedule in the final application).
> 
> 
> <snip>
> 
> > Again, improve what's there instead of creating yet another plugin...
> 
> I hope above justifies the need for gtags plugin :)

Not in my eyes, but I won't keep you from writing code :P

<snip>

Bye

-- 
Milian Wolff
mail at milianw.de
http://milianw.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20100329/e149e858/attachment.sig>