language support in kdev4

[Adam Treat]

On Monday 05 March 2007 4:43 pm, Alexander Dymo wrote:
> Hi!
> I've started porting language framework to the new KDevelop4 architecture
> and I need some help with the c++ language support now.
>
> I'll try to summarize here everything I know about the language support and
> then will ask some questions. Please correct me when I'm wrong and... for
> questions, go right to the bottom of the mail ;)

First, that's a pretty nice summary :)  I haven't been monitoring any recent 
changes, but is pretty close to what I remember...

> === How I think language support in KDev4 worked before ===
>
> 1. When the document is opened in KDevelop, the C++ language support gets
> the signal from core and calls BackgroundParser::addDocument.

Actually, there are indirections due to threading here, but basically yes.  
The language part can also add a list of documents to the background parser 
at startup.

> 2. Background parser initializes parsing by asking the active language
> support to create a parse job (LanguageSupport::createParseJob) for each
> file in the parse queue.

Right.  And the parse job can be created by either passing a url or the open 
contents of the file in ktexteditor buffer.

> 3. C++ support supplies the CPPParseJob instance when asked to and that job
> gets enqueued.

Right.

> 4. CPPParseJob becomes a parent job and adds two child jobs - PreprocessJob
> and actual ParseJob.

Right.

> 5. PreprocessJob runs the KDevelop's rpp preprocessor. When the
> preprocessor spots #include it calls PreprocessJob::sourceNeeded which asks
> language support if it can find the include (LanguageSupport::findInclude).
> If the include file is found, it is added to the background parser.

PreprocessJob _inherits_ the rpp preprocessor as well as Threadweaver's Job 
class.  When it is run, the preprocessor is run ;)  Hamish can answer this, 
but I believe that what happens is when an include is found the preprocessor 
adds the parsing of the include as a dependency.

> 6. ParseJob runs the KDevelop c++ parser, obtains the AST and uses Binder
> to fill the code model for each parsed file. AST is saved in the persistant
> hash.

Yup.   Not sure if this AST is used again, but it could be of use for the 
DUChain.

> 7. ParseJob builds the DU chain for the file. First, for each included file
> it adds the toplevel DU Context to a temporary chain. Then the
> DeclarationBuilder and UseBuilder are used. Resulting toplevel DU context
> is added to the global DU chain.
>
> 8. Toplevel DU context for a file is used to do highlighting.
>
> 9. When parsing is complete, language support is asked to insert the
> resulting code model for each parsed file into the code proxy mapping the
> url to the code model and persistent hash is filled with AST's.

About right from what I remember.

> =======================================
>
> Ok, now questions:
> 0) Are there any mistakes in my description above?
>
> 1) I can't see where the order of file parsing is set. How the background
> parser decide that it needs to parse includes before the file. Or does it
> decide that at all?

That's the dependency that PreprocessJob adds as part of sourceNeeded.  
Confirm with Hamish, but pretty sure this is how it works.

> 2) The AST lives as long as persistent hash lives (while project is
> opened), but DUContext lives in DUChain till the application shutdown. I
> can't see any DUChain::removeDocumentChain() calls. Is that ok?

Hamish?

> 3) What if we make persistent hash application-wide like DUChain, not
> project wide, so parse results can be reused even if the project is not
> opened.

I'm not sure the persistent hash is being used at all right now.

> 4) I guess both DUChain and AST needs to be saved to disk somehow, right?
> Currently they are kept only in memory.

This was a great debate.  In theory, both codemodel and DUChain can be rebuilt 
with AST and tokenstream.  But it takes CPU to create them.  And AST is 
*big*.  When I was working on this I think the hope was that the codemodel 
could be created from the ast as just a bunch of convenience methods on the 
AST/DUChain rather than storing information itself.

> 5) What would be the future of KDev3-like persistant class stores? They
> made code completion available for classes even if you did not have
> corresponding #include in the current file. Is it in our plan to implement
> completion like in KDev3?

Hamish already had beginnings of codecompletion in KDev4 from what I remember, 
and no it was _NOTHING_ like the way we did it in KDev3.  The idea was we 
just publish a model and KTextEditor would take care of the completion.  
Hamish?  Right?

> 6) We have AST, DUChain and CodeModel stored in memory in separate places.
> What if we have one common storage for them (with an option to dump to disk
> rarely used items or those items which will not change like stuff
> from /usr/include)?

Again, I think hope was that only AST and DUChain be stored.  CodeModel would 
just be convenience.  Not sure if it is feasible.

> 7) Currently we have only KDevelop::CodeItem abstracted away unlike KDev3
> where we had the whole bunch of code model classes in lib/interfaces. How
> do you think will CodeProxy::setKindFilter be enough to implement things
> like Quick Open plugin? Quick glance says it will, but maybe you see
> another problems? What would language-neutral refactoring need here? Will
> language-neutral refactoring be possible with such architecture at all?

I think this should be enough.  You want the application wide CodeModel 
interface to be very abstract.  This way we have greatest flexibilty in 
languages.

I think Quick Open could be implemented with current setup.

Jakob is going to work on refactoring this summer, I believe.  Jakob?

Hope this helps,

Adam