Implementing a general language plugin (Reposting from correct address)

Mon Apr 29 00:27:35 BST 2019

Hi,

I need to go to sleep too, but before I have time to reply more meaningfully, I just want to say that I appreciate your 
thorough response!

Cheers,

Jonathan

On pondělí 29. dubna 2019 1:08:57 CEST Francis Herne wrote:
> On Friday, 26 April 2019 22:06:03 BST Jonathan Verner wrote:
> > Hi,
> > 
> > I got addicted to semantic coloring (the thing where kdevelop colors
> > variables and their uses in separate colors) and now using other IDEs is a
> > pain :-) However, I was recently doing some frontend JavaScript work and
> > found out that kdevelop has trouble understanding modern JavaScript so I
> > had to reluctantly resort to VSCode [4].  Also, I will probably be forced
> > to write quite a lot of code in C# which, as far as I know, has no support
> > in kdevelop.  So I decided to try to sit down and write a language plugin.
> > Now there is no way that I would be able to write and *maintain* a decent
> > parser for either of the languages so the only option left to me was to use
> > the tools the languages already have.
> > 
> > However, the problem here is, that kdevelop needs language plugins to be
> > written in C++. At first I thought I would look into writing a plugin which
> > would be able to utilize the "Language Server Protocol" used by VSCode [1].
> > However, it seems that the LSP is too limited to support the cool stuff
> > kdevelop does (i.e. semantic coloring).
> 
> This isn't correct as such - all that's needed for semantic colouring is the 
> category and declaration position for each name, and those can be queried 
> through LSP.
> 
> I don't know if you've seen Emma's blog post, almost certainly the most 
> detailed consideration so far of using LSP in KDevelop:
> https://perplexinglyemma.blogspot.com/2017/06/language-servers-and-ides.html
> 
> I believe she was probably correct in the specific case of Rust, but that 
> support for LSP in KDevelop would be very useful in general.
> 
> The limitations in information returned by a particular LSP server aren't 
> relevant if you plan to implement your own. Also, anything is better than 
> nothing - there are existing LSP servers for many more languages than are ever 
> likely to have a high-quality KDevelop plugin, so even partial autocomplete/
> highlighting support is an improvement.
> 
> The more serious hurdle to overcome is that LSP's design is poorly suited to 
> feeding KDevelop's DUChain as in current language plugins - rather than 
> providing definition/use/type information in bulk, LSP is oriented toward 
> providing the final, user-facing information and actions for a code location.
> 
> In fact, LSP acts a lot like the current *querying* of the DUChain within 
> KDevelop - it supplies very similar results, but the data backing them stays 
> within the language server.
> 
> To look at it another way, KDevelop's existing DUChain and language plugins 
> could themselves make rather good LSP servers...
> 
> > So my next plan is to implement
> > something "along the lines of" LSP, but for kdevelop.  The main idea is,
> > that the plugin would connect to separate servers for each supported
> > language. The servers would then provide it with a DUChain which it would
> > feed to KDevelop. I.e. the workflow would look roughly like this
> > 
> >   1. User updates a file
> >   2. KDevelops calls my language plugin to start a Parse Job
> >   3. The plugin connects to an appropriate language server and asks it to
> > produce a DU Chain for the updated file 4. Upon receiving the DUChain it
> > updates the DUChain  KDevelop has for the file
> > 
> > I am currently experimenting with a quick and dirty implementation [5] where
> > the communication between the plugin and the server is based on gRPC ([2])
> > and ProtocolBuffers. I have a few questions:
> > 
> >   1. do you think this approach is workable (so far I didn't run into any
> > obvious roadblocks, but I am new to the codebase)
> 
> I do think it could be made to work, but I'm not sure it's the best approach 
> and there are some issues.
> 
> This is where, instead of actually answering your questions, I suggest 
> something quite different. :P
> 
> TL;DR - I believe it would be better to implement a simplified "query" API on 
> top of the DUChain, converting existing UI and non-language plugins to use it 
> instead of direct lookups, and then add support for LSP as a backend to that 
> while bypassing the DUChain entirely.
> 
> [the end goal would be to convert the DUChain and its current language plugins 
> to an LSP server, but that's far more out-of-scope]
> ---
> 
> Most existing KDevelop language plugins /read/ the DUChain as much as they 
> store things in it. To know the type of `foo.bar` and where it was declared, 
> you have to look up where `foo` was declared, and with what type, and then find 
> the declaration of `bar` within that type.
> 
> The big exception is kdev-clang, where all the declaration and typing 
> information is deduced by Clang and the language plugin is fundamentally a 
> Clang AST -> DUChain convertor.
> 
> Unless the proposed protocol is bidirectional (or the DUChain is shared cross-
> process by some other means), plugins for it can't read the DUChain 
> themselves. They must either keep their own copy of the data, reinvent an 
> equivalent framework for analyzing declarations or types, or (most likely) be 
> similar to kdev-clang and rely on an external library.
> 
> That is: plugins for it don't really /benefit/ in their own right from being 
> KDevelop-specific.
> Much as kdev-python etc. struggle with the DUChain in places (see below), all 
> of their internal analysis is based on queries to it; they wouldn't be 
> possible in anything like their current form as standalone utilities.
> In contrast, without read access a plugin has to do all the work on its own 
> and then (lossily) turn it into something the DUChain can understand.
> 
> It's important to note at this point that the DUChain is actually quite 
> restrictive, and more so the further you get from C. The implementation is 
> very elegant and performant, but it makes a lot of assumptions about how types 
> and declarations work that can be hard to map other languages onto.
> I spent a lot of time and effort trying to get kdev-python to store more things 
> usefully in the DUChain - types are objects and vice versa, any assignment 
> might be a declaration, and that's very difficult to represent.
> This also seems to be an increasing problem with C++ itself - try anything 
> involving non-trivial templates.
> 
> Given that, adding content to it through a protocol from an external plugin 
> might be quite painful. Current language plugins can define new DUChain data 
> types, are versioned in lockstep with kdevplatform so it's easy to make API 
> changes if needed, and can be frustrating anyway.
> 
> [sorry, it's midnight and I'm going to sleep. Part 2 hopefully tomorrow].
> 
> The attached file is a somewhat-relevant IRC discussion from last year. 
> Needless to say I didn't get to working on it. I might be able to this time 
> around...
> 
> > 2. my knowledge of the DU
> > Chain comes just from reading the available docs here [3] and the source
> > code of the go plugin; however, I am sometimes quite confused and I wonder,
> > if there is a tool which would output the DU Chain for a given file in a
> > "human readable" format --- that would help me very much, I think 3. I
> > haven't been able to figure out how kdevelop determines, what plugin should
> > run the parse job for a given file; so far it seems that I need to provide
> > a mime-type in the plugin json file and then kdevelop calls my plugin if
> > the mimetype matches; unfortunately, this will obviously not work if my
> > plugin wants to support different languages, ideally configured at runtime
> > based on the available language servers... Is there a way around this?
> > 
> > Anyway, if you've read this far, thanks for taking the time, even if you
> > don't have any answers :-)
> > 
> > 
> > Best,
> > 
> > Jonathan
> > 
> > 
> > [1] https://microsoft.github.io/language-server-protocol/
> > [2] https://grpc.io/
> > [3]
> > https://api.kde.org/extragear-api/kdevelop-apidocs/kdevelop/kdevplatform/ht
> > ml/index.html [4] https://code.visualstudio.com/
> > [5] It really is very dirty, but if you want to make fun of me, you can find
> > it here: https://gitlab.com/Verner/duserver
> 
> 