language support in kdev4

[Matt Rogers]

On Mar 5, 2007, at 3:43 PM, Alexander Dymo wrote:

> Hi!
> I've started porting language framework to the new KDevelop4  
> architecture
> and I need some help with the c++ language support now.
>
> I'll try to summarize here everything I know about the language  
> support and
> then will ask some questions. Please correct me when I'm wrong  
> and... for
> questions, go right to the bottom of the mail ;)
>
> === How I think language support in KDev4 worked before ===
>
> 1. When the document is opened in KDevelop, the C++ language  
> support gets the
> signal from core and calls BackgroundParser::addDocument.
>
> 2. Background parser initializes parsing by asking the active  
> language support
> to create a parse job (LanguageSupport::createParseJob) for each  
> file in the
> parse queue.
>
> 3. C++ support supplies the CPPParseJob instance when asked to and  
> that job
> gets enqueued.
>
> 4. CPPParseJob becomes a parent job and adds two child jobs -  
> PreprocessJob
> and actual ParseJob.
>
> 5. PreprocessJob runs the KDevelop's rpp preprocessor. When the  
> preprocessor
> spots #include it calls PreprocessJob::sourceNeeded which asks  
> language
> support if it can find the include (LanguageSupport::findInclude).  
> If the
> include file is found, it is added to the background parser.
>
> 6. ParseJob runs the KDevelop c++ parser, obtains the AST and uses  
> Binder to
> fill the code model for each parsed file. AST is saved in the  
> persistant hash.
>
> 7. ParseJob builds the DU chain for the file. First, for each  
> included file it
> adds the toplevel DU Context to a temporary chain. Then the  
> DeclarationBuilder
> and UseBuilder are used. Resulting toplevel DU context is added to  
> the global
> DU chain.
>
> 8. Toplevel DU context for a file is used to do highlighting.
>
> 9. When parsing is complete, language support is asked to insert  
> the resulting
> code model for each parsed file into the code proxy mapping the url  
> to the
> code model and persistent hash is filled with AST's.
>
> =======================================
>
> Ok, now questions:
> 0) Are there any mistakes in my description above?
>
> 1) I can't see where the order of file parsing is set. How the  
> background
> parser decide that it needs to parse includes before the file. Or  
> does it
> decide that at all?
>
> 2) The AST lives as long as persistent hash lives (while project is  
> opened),
> but DUContext lives in DUChain till the application shutdown. I  
> can't see any
> DUChain::removeDocumentChain() calls. Is that ok?
>
> 3) What if we make persistent hash application-wide like DUChain,  
> not project
> wide, so parse results can be reused even if the project is not  
> opened.
>
> 4) I guess both DUChain and AST needs to be saved to disk somehow,  
> right?
> Currently they are kept only in memory.
>

Something needs to be saved to disk. RAM is not large enough to hold  
kdelibs, for example, and KDevelop would crash while parsing it. This  
is what all the various catalog backends are for. The idea was to  
create a bunch of them, get some benchmarks and then choose the best  
one. IMHO, this still needs to be done, although it's fallen by the  
wayside a bit. We also need to figure out what's best stored on disk,  
since there was an idea that while we're doing a first parse on  
project opening, or whenever, we're dumping data to disk at certain  
intervals and then just querying for information from disk.


> 5) What would be the future of KDev3-like persistant class stores?  
> They made
> code completion available for classes even if you did not have  
> corresponding
> #include in the current file. Is it in our plan to implement  
> completion like
> in KDev3?
>

No. Persistant class stores are meant to be a thing of the past. For  
example, the CMake build support provides a list of places to look  
for include files so that on-the-fly parsing can be done for those  
includes. I think a cache of what's already been parsed would be a  
good idea though. However, the cache should go away on project close.


> 6) We have AST, DUChain and CodeModel stored in memory in separate  
> places.
> What if we have one common storage for them (with an option to dump  
> to disk
> rarely used items or those items which will not change like stuff
> from /usr/include)?
>

See answer to #5. If possible, we should dump to disk as much as  
possible.

> 7) Currently we have only KDevelop::CodeItem abstracted away unlike  
> KDev3
> where we had the whole bunch of code model classes in lib/ 
> interfaces. How do
> you think will CodeProxy::setKindFilter be enough to implement  
> things like
> Quick Open plugin? Quick glance says it will, but maybe you see  
> another
> problems? What would language-neutral refactoring need here? Will
> language-neutral refactoring be possible with such architecture at  
> all?
>

That's what we need Jakob to figure out. We won't know until somebody  
tries it.
--
Matt