On Memory consumption of KDevelop-PG
zwabel at googlemail.com
Sat Dec 19 17:45:03 UTC 2009
Am Samstag 19 Dezember 2009 17:36:59 schrieb Alexander Dymo:
> субота, 19-гру-2009 11:49:47 Andreas Pakulat ви написали:
> > IMHO its no big deal if the parser needs some 200M for parsing a single
> > file as long as afterwards the memory is free again for re-use. In
> > particular it must not be fragmented (which should already work due to
> > usage of the memory pool). For how to make the AST smaller see further
> > down on your "compress the AST" question.
> Well, but now huge parser memory consumption is an issue (at least for
> C++). I have 3.5M C file which is parsed into 550M AST.
> 550M isn't that small amount of memory...
> And that's just one file. Try parsing Ruby language sources - you'll end up
> using 3G of memory for parsing. That's definitely wrong. Not everybody has
> this much of memory.
> There're actually bugs in c++ parser where it creates AST nodes which
> aren't used (added to the tree). I have a patch (not public yet, still
> breaks some duchain tests) for that which reduces the memory consumption
> in my test case from 550M to 400M. But that's all I can do without doing
> AST compression or removing members from AST classes.
There is a few things we can do in future that would have a huge impact:
- Replace size_t with uint everywhere within the parser
- Move the "AST* parent" and "DUContext* ducontext" members out of the AST,
into a separate map in ParseSession, as they are not needed in most cases.
- Do the same thing with the "comments" member of CommentAST
Together, these would probably reduce the size of the AST by 2/3.
However I consider this all that important atm, because the performance needs
to be considered carefully when doing some of these changes, and KDevelop4
right now anyway needs a fairly modern machine, which makes the memory-
requirements not all that fatal. The more important issues right now are
stability and usability.
And btw. how did you measure those 550MB? It sounds very much..
More information about the KDevelop-devel