Memory consumption with large projects

[Milian Wolff]

On Thursday, 26. November 2009 09:55:55 David Nolden wrote:
> Am Donnerstag 26 November 2009 00:09:55 schrieb Milian Wolff:
> > On Wednesday 25 November 2009 23:50:04 Alexander Dymo wrote:
> > > I tried to open a project with roughly 500 000 lines of C code (about
> > > 180M on disk) and got some strange issues with memory consumption.
> > >
> > > During parsing (no persistant storage used):
> > > - 3.8G used right after parser is started and before it showed 1% in
> > > the progress bar
> > > - it then looked like that memory was free'd and KDevelop started
> > > actually to parse
> > > - 2.2G occupied after parsing
> > >
> > > During loading from persistant store:
> > > - just 200M was used (at any time)
> > >
> > >
> > > Do those numbers make sense? I'd be most curious to find out why KDev
> > > takes so much memory.
> >
> > Hey Adymo! Off to fixing bugs in Ruby again? :)
> >
> > This behavior is known. Afaik it is suspected to be due to (at least in
> >  some degree) the kernel/whoever not coping well with lots of small frees
> >  => memory fragmentation.
> 
> Me personally I have never suspected this. ;-)

I'm the young padawan in this regard ;-)

Some users in #kdevelop stated it, and I read about it on some places in the 
internet. Just look at e.g. the docu of the Boost pool.

And especially if you consider that valgrind does not find any definite leaks. 
And afaik the last time I checked the number reported by valgrind was far too 
low, compared to what ksysguard reported. Hence my impression that 
memoryfragmentation did take place.
 
> > We should probably start to use memory pools here and there, at least
> > that was the "solution" I have in my memory.
> 
> We should find out what exactly is eating up the memory that much, massif
> should be able to tell us. I've already fixed quite a few memory hogs, but
> none of them has been 'the one' until now.


-- 
Milian Wolff
mail at milianw.de
http://milianw.de