Memory consumption with large projects

[Alexander Dymo]

четвер, 26-лис-2009 10:55:55 David Nolden ви написали:
> We should find out what exactly is eating up the memory that much, massif
> should be able to tell us. I've already fixed quite a few memory hogs, but
> none of them has been 'the one' until now.

I ran KDevelop under massif... or better to say I ran it until it crashed 
(most likely killed by OOM).

Results attached (stripped, only peak snapshot data before crash).


I could only find one obvious problem with tokenizer - token stream is 
increased in size in 2x increments. That usually works, but for projects where 
token stream, for example, would hit 300M, next increment will increase its 
size to 600M which is not what we want to do.

I'd say the size increment should be 2x, but not larger than some constant, 
say 20M.


But anyway, token stream takes only 670M, other 2G are taken by AST nodes 
(mostly SimpleTypeSpecifierAST's). Given that the project is huge (as I've 
said 500k sloc), that might be expected. Unfortunatelly massif doesn't tell 
the number of allocations, so I can't tell whether taking 670M by 
SimpleTypeSpecifierAST's sounds right or not.

Correct me if I wrong, but we don't store AST's in persistant storage. So when 
loading from it we just load duchain, right? That might explain the huge 
difference between 200M (persistant store used) and 2.9G (fresh reparse) - the 
AST takes all the memory.

If my assumption is correct, then we might want to:
1) reduce the size of AST nodes (haven't actually looked whether that's 
possible)
2) split the parsing of large projects into N subsequent steps (for example, 
don't allow to parse more than 100 files at one go). also not sure whether 
that's sensible, just don't have enough cpp parser/duchain knowledge
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ms_print_out.txt.bz2
Type: application/x-bzip
Size: 6741 bytes
Desc: not available
URL: <http://mail.kde.org/pipermail/kdevelop-devel/attachments/20091127/bdf2be21/attachment.bz2>