[Kst] memory checks...
netterfield at astro.utoronto.ca
Thu May 27 22:51:43 CEST 2004
/proc/meminfo appears to be the only thing under linux.
linux has a command 'free' which uses /proc/meminfo.
We'll have to add some configure tests for /proc/meminfo of the correct
On May 27, 2004 03:28 pm, George Staikos wrote:
> On May 27, 2004 14:38, Barth Netterfield wrote:
> > A very real possibility which I have done myself, and made bad things
> > happen.
> > omega% kst -f 0 /data/rawdir/B2K2/ -y bolo1 -y bolo2 -y bolo3 -y bolo4 -y
> > bolo5 -y bolo6 -y bolo7 -m 2
> > I just asked it to read 7x54million sample vectors (+ INDEX). I meant to
> > add -n 1000000, but forgot. Each vector is 216 MB, so my 512 MB machine
> > is going to run out of memory and start swapping, then run out of swap
> > and die.
> > The *only* questions are:
> > i) under linux, is it possible?
> > ii) if so, how.
> > For (i) the answer is clearly "yes". I can run 'top' and see how much
> > free ram there is, how much ram is being used for cache, and how much
> > swap is free. Top gets this information, so we can get this information.
> > Getting this information before the malloc could allow us to make some
> > intelligent decisions about what to do. (ie, I probably don't want to
> > allocate more than free+cache - if I do my machine will likely be pretty
> > much useless or will just die).
> The problem is that in your case, there is no such thing as "the
> malloc". There are 7 mallocs, each of which is reasonable, but combined,
> become deadly. So we need to do even more in-depth calculations of how
> much ram is free (which we don't really know since it may have been
> overcommitted to some other app already), and we have to track all of our
> allocations and frees. This will not work for situations of running
> multiple Kst instances too.
> > This of course is not bullet proof, (eg, memory gets allocated or freed
> > by some other program/thread in between checking and allocating) but this
> > will be relativly rare, and I bet it would catch the vast majority of
> > problems. Enough problems would be caught to be easily worth the effort.
> What we need then is an algorithm that can guess at how much we can
> safely allocate.
More information about the Kst