language support in kdev4

[Alexander Dymo]

Hi!
I've started porting language framework to the new KDevelop4 architecture
and I need some help with the c++ language support now.

I'll try to summarize here everything I know about the language support and
then will ask some questions. Please correct me when I'm wrong and... for
questions, go right to the bottom of the mail ;)

=== How I think language support in KDev4 worked before ===

1. When the document is opened in KDevelop, the C++ language support gets the
signal from core and calls BackgroundParser::addDocument.

2. Background parser initializes parsing by asking the active language support
to create a parse job (LanguageSupport::createParseJob) for each file in the
parse queue.

3. C++ support supplies the CPPParseJob instance when asked to and that job
gets enqueued.

4. CPPParseJob becomes a parent job and adds two child jobs - PreprocessJob
and actual ParseJob.

5. PreprocessJob runs the KDevelop's rpp preprocessor. When the preprocessor
spots #include it calls PreprocessJob::sourceNeeded which asks language
support if it can find the include (LanguageSupport::findInclude). If the
include file is found, it is added to the background parser.

6. ParseJob runs the KDevelop c++ parser, obtains the AST and uses Binder to
fill the code model for each parsed file. AST is saved in the persistant hash.

7. ParseJob builds the DU chain for the file. First, for each included file it
adds the toplevel DU Context to a temporary chain. Then the DeclarationBuilder
and UseBuilder are used. Resulting toplevel DU context is added to the global
DU chain.

8. Toplevel DU context for a file is used to do highlighting.

9. When parsing is complete, language support is asked to insert the resulting
code model for each parsed file into the code proxy mapping the url to the
code model and persistent hash is filled with AST's.

=======================================

Ok, now questions:
0) Are there any mistakes in my description above?

1) I can't see where the order of file parsing is set. How the background
parser decide that it needs to parse includes before the file. Or does it
decide that at all?

2) The AST lives as long as persistent hash lives (while project is opened),
but DUContext lives in DUChain till the application shutdown. I can't see any
DUChain::removeDocumentChain() calls. Is that ok?

3) What if we make persistent hash application-wide like DUChain, not project
wide, so parse results can be reused even if the project is not opened.

4) I guess both DUChain and AST needs to be saved to disk somehow, right?
Currently they are kept only in memory.

5) What would be the future of KDev3-like persistant class stores? They made
code completion available for classes even if you did not have corresponding
#include in the current file. Is it in our plan to implement completion like
in KDev3?

6) We have AST, DUChain and CodeModel stored in memory in separate places.
What if we have one common storage for them (with an option to dump to disk
rarely used items or those items which will not change like stuff
from /usr/include)?

7) Currently we have only KDevelop::CodeItem abstracted away unlike KDev3
where we had the whole bunch of code model classes in lib/interfaces. How do
you think will CodeProxy::setKindFilter be enough to implement things like
Quick Open plugin? Quick glance says it will, but maybe you see another 
problems? What would language-neutral refactoring need here? Will
language-neutral refactoring be possible with such architecture at all?