<div dir="ltr">And again I reply only to the last poster, instead of the kdevelop-devel list. Original message follows...<br><div><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Olivier J. G.</b> <span dir="ltr"><<a href="mailto:olivier.jg@gmail.com">olivier.jg@gmail.com</a>></span><br>
Date: Thu, May 9, 2013 at 5:27 PM<br>Subject: Re: Clang<br>To: Milian Wolff <<a href="mailto:mail@milianw.de">mail@milianw.de</a>><br><br><br><div dir="ltr"><div><div><div>I didn't spend much time getting stuff working well once it could apparently be done, as I was focused on finding if it was a reasonable thing to do. That said, it can build declarations, contexts and uses.<br>
<br></div>Using libclang would make sense if you just want access to the AST, I mainly didn't use it because I assumed I'd have more control over the parsing process using the C++ API.<br><br></div>Using the current C++ support is, as Milian mentioned, not going to help. Making the parser support C++11 is fairly easy (and it mostly does). Our weakness is currently in the semantic analysis, especially support for templates. The big win with Clang is reusing the semantic analysis. My thinking here is especially to not go the TemplateDeclaration route, and instead put all the template information in the type system, i.e. UnsureType (I'd be interested to hear from David about his take on this though). This allows us to better support writing code which uses template parameters, at the very least.<br>
<br></div><div>It's important to note that the AST generated by clang is fully 'annotated' with everything you'd ever need to know to make a duchain. It's trivial to go from a Clang AST to a duchain structure. After that, it's only a matter of subclassing the duchain classes to provide more of the information already collected in the Clang AST.<br>
<br></div><div>Our problem is that Clang creates ASTs for translation units, and we want/need ASTs for files (lets ignore the preprocessor elephant in the room, for now).<br>Currently, using Clang, when a cpp file is edited, you have to regenerate the entire translation unit for that file, which could involve reparsing a lot of headers. This is why you often hear about pch files when people talk about clang-complete plugins. We have the TopDUContext, meant to represent a file, which is the solution to clang being unusable for update-as-you-type situations. In order to use that though, we need Clang to be able to operate on individual files, and not expand includes.<br>
<br></div><div>Now, to not ignore the preprocessor elephant, this is not an easy problem, and may be as hard as the holy grail of C++ compilers: incremental compilation. KDevelop solves this problem with the ParsingEnvironment, which I'm not familiar enough with to say anything about. In short, the same C++ file could be represented by any number of different DUChain structures depending on the state of the preprocessor by the time it is reached. From an IDE perspective, the only way to represent this concept is to allow the end-user to select the current preprocessor state of a file from within the IDE.<br>
</div><div><br></div><div>Obviously Clang cannot give us its perfect "AST" with everything known, and all types and functions resolved and cross-referenced, while only working with a single file. The solution to this from a Clang perspective might be to have it create lower-level, per-file ASTs, each of which would have the information: "given preprocessor state A and file B, the AST for this file is X and the outgoing preprocessor state is Y". From there you could hopefully generate semantic information for any subset of the entire TU much more cheaply, on demand. Possibly I need to be hit with a clue bat, and possibly this is incremental parsing, but in any case I can't imagine that this wouldn't mean rewriting much of Clang.<br>
<br></div><div>Here's the real fun though... I just reinvented the DUChain and put it in Clang. All those ASTs need to be cached, otherwise they're completely useless, and basically all that KDevelop would do is copy the information into the language agnostic DUChain, which would cache it again. The reason for this apparent paradox is that creating a DUChain requires semantic analysis, and the DUChain expects to shoulder the cost of keeping that analysis in memory. Clang provides an interesting opportunity here to take advantage of some incredibly complicated semantic analysis done for free, where most language plugins have to do all that themselves starting with just a basic AST (not in the Clang sense). The problem is that the semantic analysis has to feed into itself. Once I've duchainified foo.h, I use that information to duchainify foo.cpp. Therefore the semantic analyser must be the cacher as well.<br>
<br></div><div>So, to summarize, there are two ways to use Clang:<br></div><div>1. Get into the C++ API, use it as a preprocessor/parser, and based on that do semantic analysis and build the duchain<br></div><div>2. Write some manner of reasonable caching for Clang's completed "AST", copy into duchain<br>
</div><div class="gmail_extra"><br></div><div class="gmail_extra">I hope someone has identified a flaw in my analysis here, and hits me with a clue bat, because this isn't very inspiring.<br><br></div><div class="gmail_extra">
-Olivier JG<br><br></div><div class="gmail_extra">PS: What I mentioned is not the only way to cache Clang ASTs, there was a discussion/proposal on the Clang ML a while back about a "Clang Daemon", which would be "good enough" for our purposes. Unfortunately, as far as I know, no lines of code have been written.<br>
<br>Footnote: The "semantic analysis" I keep going on about is knowing what uses refer to what declarations.<br></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div class="h5">On Sun, May 5, 2013 at 4:14 PM, Milian Wolff <span dir="ltr"><<a href="mailto:mail@milianw.de" target="_blank">mail@milianw.de</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="h5"><div><div>On Sunday 05 May 2013 21:52:59 Alexandre Courbot wrote:<br>
> On Sun, May 5, 2013 at 8:36 PM, Aleix Pol <<a href="mailto:aleixpol@kde.org" target="_blank">aleixpol@kde.org</a>> wrote:<br>
> > I'm unsure if it doesn't make sense, but has it been considered forking<br>
> > the<br>
> > current c++ implementation and just replacing the current parser (and even<br>
> > the AST) for Clang's?<br>
> ><br>
> > Maybe it's easier if we do a 1:1 port...<br>
><br>
> That's what I did in my own fork. Actually I just replaced the<br>
> parser/AST part with CLang's (in a CLangParseJob class) and build the<br>
> DUChain from there, using the helper functions available in<br>
> cpplanguagesupport. That seems to work and I agree it's probably the<br>
> shortest path to CLang parsing.<br>
<br>
</div></div>Parsing alone is not helpful. The most brittle part of our C++ support, and<br>
also the part that requires the most work, is the semantic analysis. So we<br>
really want to use the annotated/analyzed AST that clang provides and throw<br>
away as much of our code as possible (long term).<br>
<br>
Anyhow, I still didn't have time to look into it. All I know is from speaking<br>
with Olivier about it. I'll definitely look into this eventually though after<br>
this semester is finally done...<br>
<div><div><br>
Cheers<br>
--<br>
Milian Wolff<br>
<a href="mailto:mail@milianw.de" target="_blank">mail@milianw.de</a><br>
<a href="http://milianw.de" target="_blank">http://milianw.de</a></div></div><br></div></div><div class="im">_______________________________________________<br>
KDevelop-devel mailing list<br>
<a href="mailto:KDevelop-devel@kde.org" target="_blank">KDevelop-devel@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/kdevelop-devel" target="_blank">https://mail.kde.org/mailman/listinfo/kdevelop-devel</a><br>
<br></div></blockquote></div><br></div></div>
</div><br></div></div>