[Kst] memory checks...

[George Staikos]

On May 27, 2004 14:38, Barth Netterfield wrote:
> A very real possibility which I have done myself, and made bad things
> happen.
>
> omega% kst -f 0 /data/rawdir/B2K2/ -y bolo1 -y bolo2 -y bolo3 -y bolo4 -y
> bolo5 -y bolo6 -y bolo7 -m 2
>
> I just asked it to read 7x54million sample vectors (+ INDEX).  I meant to
> add -n 1000000, but forgot.  Each vector is 216 MB, so my 512 MB machine is
> going to run out of memory and start swapping, then run out of swap and
> die.
[...]
> The *only* questions are:
> 	i) under linux, is it possible?
> 	ii) if so, how.
>
> For (i) the answer is clearly "yes".  I can run 'top' and see how much free
> ram there is, how much ram is being used for cache, and how much swap is
> free.  Top gets this information, so we can get this information.  Getting
> this information before the malloc could allow us to make some intelligent
> decisions about what to do.  (ie, I probably don't want to allocate more
> than free+cache - if I do my machine will likely be pretty much useless or
> will just die).

   The problem is that in your case, there is no such thing as "the malloc".  
There are 7 mallocs, each of which is reasonable, but combined, become 
deadly.  So we need to do even more in-depth calculations of how much ram is 
free (which we don't really know since it may have been overcommitted to some 
other app already), and we have to track all of our allocations and frees.  
This will not work for situations of running multiple Kst instances too.

> This of course is not bullet proof, (eg, memory gets allocated or freed by
> some other program/thread in between checking and allocating) but this will
> be relativly rare, and I bet it would catch the vast majority of problems.
> Enough problems would be caught to be easily worth the effort.

   What we need then is an algorithm that can guess at how much we can safely 
allocate.

-- 
George Staikos
KDE Developer			http://www.kde.org/
Staikos Computing Services Inc.	http://www.staikos.net/