GSoC 2010: Making KDevelop suitable for large C projects

[Nitin Gupta]

Hi,

On 03/29/2010 06:03 PM, Milian Wolff wrote:
> Nitin Gupta, 27.03.2010:
> 
>> For past few months, I have been working on a simple IDE (called KXref)
>> based on Sublime UI framework, to allow working with large C based
>> projects (Linux kernel in particular): http://code.google.com/p/kxref/
>>
>> However, I soon realized that it will be much more useful and productive if
>> all this can be developed as plugins for upcoming KDevelop4. My initial
>> attempt to get it working for Linux kernel failed badly primarily due to
>> lack of cscope, or better GNU global, integration (built-in background
>> parser is too slow for such large projects). There are many other factors
>> which make it unsuitable for kernel projects (as detailed below).
>>
>> Unlike popular belief, an IDE for kernel projects (IMHO) is highly
>> desirable. Trying to understand such huge and complex codebases is almost
>> impossible otherwise. Plain vim+ctags is often not sufficient.
>>
>> I'm planning to submit this as a GSoC proposal. I would be thankful for any
>> feedback.
> 
> My personal opinion is: One can integrate each and every tool there is for 
> KDevelop, but I'd much prefer improving the performance of the build in parser 
> instead of using yet another tool for the job... 
> 
> I found quite a few places to optimize over the weeks, and I bet there are 
> more.
> 

Built-in parser seems to be slow beyond repair. Following is test summary:

 * Project 1 (stripped down linux kernel tree)
    - Size: ~18000 c, h files
    - Time for backgroud parser to complete till 10%: 18 mins
    - I closed KDevelop at this time. But at this rate, it would have taken 3 hrs!!
    - Time taken by GNU global for 100% completion (gtags): 9 seconds.
    - Restarted KDevelop. Now the parser starts from beginning! Expected it to start from 10%.

 * Project 2 (stripped down above tree even further)
    - Size: ~600 c, h files
    - Time taken by in-built parser (100% completion): 1 min 30 seconds
    - Time taken by gtags: 0.6 seconds
    - Verified that the in-built parser created database in ~/.kdevduchain however each time
      KDevelop is restarted, it seems to *recreate* the same database. It seems only dumping
      the cross reference db is currently supported and not really reusing it.

I have not yet started digging into code but I guess above stark difference in gtags vs in-built
parser speed is due to much more contextual information gathered by the latter (like usage of a
local variables withing functions etc), but now quite sure.

Considering above, I think it makes sense to implement GNU Global (gtags) plugin to handle large
(especially C) projects. At the same time, use in-built parser to provide rich local context
information -- like highlighting usage of a local variable within its scope -- for currently opened
files only.


GNU Global claims to support C, C++, Yacc, Java and PHP4 so this plugin might be useful for large
projects based on all these languages but I'm not really sure how good is gtags support for languages
other than C.

Apart from faster parsing, essential components for supporting large C projects are:
 - Graph Viewer: Use information from gtags query to quickly generate call graphs
    - Check if existing graph viewer can be extended to use gtags or a new one needs to be developed.
 - Query Viewer: Again use gtags for queries and show results as generated by gtags
    - Current 'Code Browser' window does not show query history. A tabbed query result window is needed
      to maintain query history (very important).
 - Generic C template(s)
 - Linux kernel specific template(s)

I think all above is doable within GSoC timelines (I will include project schedule in the final application).


<snip>
> 
> Again, improve what's there instead of creating yet another plugin...
> 

I hope above justifies the need for gtags plugin :)


>>  * SCM support:
>>  - Git support exists in kdevelop4-extra-plugins but not sure how robust it
>> is. For good Linux kernel project support, need to make sure that this
>> works well.
> 
> This will be done (hopefully) by another GSOC student. I think that alone is 
> enough work for a few weeks.
>

Nice to know Git support is getting attention in parallel!

 
>>  - Better file history navigation as supported by p4v for Perforce (as a
>> KDevelop plugin): http://www.perforce.com/perforce/products/p4v.html
>>
>>  * Built-in 'oops' (kernel crash messages) interpreter:
>>  - Transform plain oops message into references to functions shown in
>> backtrace.
>>  - Interpret stack dump into function parameters.
> 
> Really good idea, should also be applicable to GDB backtraces.
>  

Yes. Also, parsing oops messages directly from kerneloops.org will be helpful.
Example: http://kerneloops.org/raw.php?rawid=2801720
The backtrace from this could be downloaded, parsed and linked to kernel code.


>>  * Integration with SystemTap (http://sourceware.org/systemtap/wiki):
>>  - Explore existing tap points.
>>  - Easily create new one with auto-generated scripts (provide library of
>> commonly used scripts).
>>  - Visualizers for various kinds of stats exported by SystemTap.
> 
> I really think this is far too much for a single GSOC. Also we have the 
> problem with lack of mentors, but maybe someone else would be willing to help?
> 

Yes, SystemTap integration would really be far too much for a single GSoC project.
Still, all other aspects (as detailed above) are, I believe, doable within GSoC.


> I still would really appreciate at least a few parts of this proposal to be 
> put into life. Looking forward to your first patches :)
>

Much of my existing effort got wasted into developing an IDE called KXref[1] from
scratch (stupid me) which can currently handle, at least, opening multiple files
at once and other basic functionality.
Now, since the KDevelop plugin approach has been decided, I think the work is now
well defined and hopefully in the right direction :)


Thanks for your comments and feedback.
Nitin


[1] http://code.google.com/p/kxref/