GSoC 2010: Making KDevelop suitable for large C projects

[Nitin Gupta]

On 03/30/2010 01:31 AM, Milian Wolff wrote:
> On Monday 29 March 2010 21:48:23 Nitin Gupta wrote:
>> On 03/30/2010 12:37 AM, Andreas Pakulat wrote:
>>> On 29.03.10 23:16:06, Nitin Gupta wrote:
>>>> On 03/29/2010 06:03 PM, Milian Wolff wrote:
>>>>> Nitin Gupta, 27.03.2010:
>>>>>> For past few months, I have been working on a simple IDE (called
>>>>>> KXref) based on Sublime UI framework, to allow working with large C
>>>>>> based projects (Linux kernel in particular):
>>>>>> http://code.google.com/p/kxref/
>>>>>>
>>>>>> However, I soon realized that it will be much more useful and
>>>>>> productive if all this can be developed as plugins for upcoming
>>>>>> KDevelop4. My initial attempt to get it working for Linux kernel
>>>>>> failed badly primarily due to lack of cscope, or better GNU global,
>>>>>> integration (built-in background parser is too slow for such large
>>>>>> projects). There are many other factors which make it unsuitable for
>>>>>> kernel projects (as detailed below).
>>>>>>
>>>>>> Unlike popular belief, an IDE for kernel projects (IMHO) is highly
>>>>>> desirable. Trying to understand such huge and complex codebases is
>>>>>> almost impossible otherwise. Plain vim+ctags is often not sufficient.
>>>>>>
>>>>>> I'm planning to submit this as a GSoC proposal. I would be thankful
>>>>>> for any feedback.
>>>>>
>>>>> My personal opinion is: One can integrate each and every tool there is
>>>>> for KDevelop, but I'd much prefer improving the performance of the
>>>>> build in parser instead of using yet another tool for the job...
>>>>>
>>>>> I found quite a few places to optimize over the weeks, and I bet there
>>>>> are more.
>>>>
>>>> Built-in parser seems to be slow beyond repair. Following is test 
> summary:
>>> I suggest you back that statement up by some knowledge about the
>>> codebase. Making such blanket statements won't get you much friends
>>> around here ;)
>>
>> Yes, I should have gathered proper profiling data but the numbers are still
>> very disappointing. Argument that its all done in background with
>> preference given to opened files isn't too nice -- with large C project
>> containing 20k - 30k files, the parser (in its current form) can take
>> hours pumping CPU at nearly 100%. Maybe improving in-built parser speed is
>> the way to go but it was too tempting for me to go for already optimized
>> solutions like gtags even if they could not provide as much of information
>> as in-built parser.
> 
> If gtags would provide all required to fill the DUChain, try to use it. You can 
> simply create a new alternative language plugin that fills the data in the 
> DUChain from gtags. As I said: it's all possible.
>

After feedback from you all, I'm now thinking of simply dumping on all this gtags
stuff. It was especially good to know that we have comprehensive parsing vs speed
switch in the parser. I just tried that but reducing:
  "Minimum project size for simplified parsing" to 1000 so that the project does use
faster parsing but that does not seems to be any faster.

Anyways, the point now is that it will be really exciting to see where the bottlenecks
are. One simple observation from gtags is that it never uses multiple cores while the
built-in parser drives both CPUs (dual core machine) to nearly 50%-60% constantly. Even
then the parser (even simplified parsing) is so much slower. I think optimizing this
parser is worth a project in itself.

 
>> Maybe we should have options for in-built parser like: more comprehensive
>> parsing vs speed. I think this will quickly make KDevelop suitable for
>> large projects while it is gradually optimized.
> 
> Also exists already, might have to push it further.
> 
> <...>
> 
>>>> Apart from faster parsing, essential components for supporting large C 
> projects are:
>>>>  - Graph Viewer: Use information from gtags query to quickly generate
>>>>  call graphs
>>>>  
>>>>     - Check if existing graph viewer can be extended to use gtags or a
>>>>     new one needs to be developed.
>>>
>>> This is available as playground plugin already based on duchain,
>>> automatically working for all language plugins we have.
>>>
>>>>  - Query Viewer: Again use gtags for queries and show results as
>>>>  generated by gtags
>>>>  
>>>>     - Current 'Code Browser' window does not show query history. A
>>>>     tabbed query result window is needed
>>>>     
>>>>       to maintain query history (very important).
>>>
>>> What exactly do you mean with "query"?
>>
>> I meant queries like "find callers of this function", "find definition of
>> such function" etc.
> 
> But you know that they already exist with the DUChain? And that there is no 
> way around the DUChain since otherwise you'd need a whole bunch of new plugins 
> making it essentially pointless to have such a nice abstraction like the 
> DUChain.
> 
> Look at QtCreator, it's blazingly fast but otoh doesn't have many feature that 
> KDevelop has and is also very reliant on the AST, meaning: no language 
> Independence at all.
> 
> 

Got your point. I will try dig through DUchain code, learn userspace profilers and
maybe understand and fix the bottlenecks.

Thanks,
Nitin