language support in kdev4

[Hamish Rodda]

On Tuesday 06 March 2007 08:43, Alexander Dymo wrote:
> Hi!
> I've started porting language framework to the new KDevelop4 architecture
> and I need some help with the c++ language support now.
>
> I'll try to summarize here everything I know about the language support and
> then will ask some questions. Please correct me when I'm wrong and... for
> questions, go right to the bottom of the mail ;)
>
> === How I think language support in KDev4 worked before ===
>
> 1. When the document is opened in KDevelop, the C++ language support gets
> the signal from core and calls BackgroundParser::addDocument.
>
> 2. Background parser initializes parsing by asking the active language
> support to create a parse job (LanguageSupport::createParseJob) for each
> file in the parse queue.
>
> 3. C++ support supplies the CPPParseJob instance when asked to and that job
> gets enqueued.
>
> 4. CPPParseJob becomes a parent job and adds two child jobs - PreprocessJob
> and actual ParseJob.
>
> 5. PreprocessJob runs the KDevelop's rpp preprocessor. When the
> preprocessor spots #include it calls PreprocessJob::sourceNeeded which asks
> language support if it can find the include (LanguageSupport::findInclude).
> If the include file is found, it is added to the background parser.

There is a problem here... we need to make the preprocessor interruptable, if 
possible, so that the job can sleep while the include is processed 
separately, if not already processed in memory.  Currently it only includes 
if it's already processed (iirc).

There is another problem too... currently I save an intemediate version of the 
preprocessed code for speed when #including, but it's not 100% accurate, 
probably I need to store a less processed version.  It's not a big problem 
though, I think.

> 6. ParseJob runs the KDevelop c++ parser, obtains the AST and uses Binder
> to fill the code model for each parsed file. AST is saved in the persistant
> hash.

The Binder is hopefully going to be superceeded by the DUChain, once the code 
model is adapted to extract all information from the DUChain.  Then we don't 
need a separate code model.

> 7. ParseJob builds the DU chain for the file. First, for each included file
> it adds the toplevel DU Context to a temporary chain. Then the
> DeclarationBuilder and UseBuilder are used. Resulting toplevel DU context
> is added to the global DU chain.

There's nothing temporary about the chain.  It's only created once per file, 
then reused.  One day, hopefully even incrementally rebuilt.

> 8. Toplevel DU context for a file is used to do highlighting.
>
> 9. When parsing is complete, language support is asked to insert the
> resulting code model for each parsed file into the code proxy mapping the
> url to the code model and persistent hash is filled with AST's.
>
> =======================================
>
> Ok, now questions:
> 0) Are there any mistakes in my description above?
>
> 1) I can't see where the order of file parsing is set. How the background
> parser decide that it needs to parse includes before the file. Or does it
> decide that at all?

Well, when I used threadweaver to set up the dependencies, threadweaver would 
crash.  I don't know if that's been fixed (you could uncomment to find out, I 
guess).

Then there's the problem of detecting cyclic dependencies and dealing with 
them.

> 2) The AST lives as long as persistent hash lives (while project is
> opened), but DUContext lives in DUChain till the application shutdown. I
> can't see any DUChain::removeDocumentChain() calls. Is that ok?

Ah, I _think_ the chain might be cleared on project close.  Not sure though.  
That's what should happen, anyway.

> 3) What if we make persistent hash application-wide like DUChain, not
> project wide, so parse results can be reused even if the project is not
> opened.

You'd get completions etc. from other projects in your active project... no, I 
think they should be per project.

> 4) I guess both DUChain and AST needs to be saved to disk somehow, right?
> Currently they are kept only in memory.

The idea is to one day be able to serialize the DUChain to disk, and throw out 
the AST and regenerate as needed.  The AST needs to be kept for when we do 
incremental parsing.  The DUChain could replace the persistant class store 
mechanism in kdev4.

> 5) What would be the future of KDev3-like persistant class stores? They
> made code completion available for classes even if you did not have
> corresponding #include in the current file. Is it in our plan to implement
> completion like in KDev3?

If the DUChain can be serialized (should be no problem), this should replace 
the persistant class store (as long as it isn't too big, haven't looked into 
that).  At least, we should serialize globally accessible stuff.

> 6) We have AST, DUChain and CodeModel stored in memory in separate places.
> What if we have one common storage for them (with an option to dump to disk
> rarely used items or those items which will not change like stuff
> from /usr/include)?

AST: throw away as needed and recreate
DUChain: serialize and restore
CodeModel: hopefully to be superseeded by DUChain

> 7) Currently we have only KDevelop::CodeItem abstracted away unlike KDev3
> where we had the whole bunch of code model classes in lib/interfaces. How
> do you think will CodeProxy::setKindFilter be enough to implement things
> like Quick Open plugin? Quick glance says it will, but maybe you see
> another problems? What would language-neutral refactoring need here? Will
> language-neutral refactoring be possible with such architecture at all?

For neutral refactoring I think we need to make a generic interface to the 
DUChain (eg. KDevDUChain with C++ stuff extracted to a subclass, CPPDUChain) 
and work from there.

Cheers,
Hamish.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20070306/23962a85/attachment.sig>