On Memory consumption of KDevelop-PG

[David Nolden]

Am Samstag 19 Dezember 2009 17:36:59 schrieb Alexander Dymo:
> субота, 19-гру-2009 11:49:47 Andreas Pakulat ви написали:
> > IMHO its no big deal if the parser needs some 200M for parsing a single
> > file as long as afterwards the memory is free again for re-use. In
> > particular it must not be fragmented (which should already work due to
> > usage of the memory pool). For how to make the AST smaller see further
> > down on your "compress the AST" question.
> 
> Well, but now huge parser memory consumption is an issue (at least for
>  C++). I have 3.5M C file which is parsed into 550M AST.
> 550M isn't that small amount of memory...
> 
> And that's just one file. Try parsing Ruby language sources - you'll end up
> using 3G of memory for parsing. That's definitely wrong. Not everybody has
> this much of memory.
> 
> There're actually bugs in c++ parser where it creates AST nodes which
>  aren't used (added to the tree). I have a patch (not public yet, still
>  breaks some duchain tests) for that which reduces the memory consumption
>  in my test case from 550M to 400M. But that's all I can do without doing
>  AST compression or removing members from AST classes.

There is a few things we can do in future that would have a huge impact:
- Replace size_t with uint everywhere within the parser
- Move the "AST* parent" and "DUContext* ducontext" members out of the AST, 
into a separate map in ParseSession, as they are not needed in most cases.
- Do the same thing with the "comments" member of CommentAST

Together, these would probably reduce the size of the AST by 2/3.

However I consider this all that important atm, because the performance needs 
to be considered carefully when doing some of these changes, and KDevelop4 
right now anyway needs a fairly modern machine, which makes the memory-
requirements not all that fatal. The more important issues right now are 
stability and usability.

And btw. how did you measure those 550MB? It sounds very much..