[kstars-devel] analysis of memory consumption
Heiko Evermann
Heiko.Evermann at gmx.de
Mon Sep 1 22:54:19 CEST 2003
Hi everyone,
some more information about the memory consumption:
I tried adding every star twice inKStarsData::processStar by using
> StarObject *o = new StarObject(r, d, mag, name, gname, SpType,
pmra, pmdec, plx, mult, var );
> starList.append(o);
> StarObject *p = new StarObject(r, d, mag, name, gname, SpType,
pmra, pmdec, plx, mult, var );
> starList.append(p);
When I load all 126000 stars from the catalog, taking into consideration
that loading 0 stars leads to 40MB memory consumption, I get:
117 MB for 126000 stars => 77 MB for the stars => 625 bytes per star
181 MB for 252000 stars => 141 MB for the stars => 573 bytes per star.
A star object on its own uses 364 bytes + string lengths +
administrative overhead in the lists. So we need about 200 bytes
overhead for names and list management. This cannot be right.
I further checked the string lenghts in
>StarObject::StarObject( dms r, dms d, float m, QString n, QString n2,
QString sptype,
> double pmra, double pmdec, double par, bool mult, bool var )
using
>
> static lenN = 0;
> static lenN2 = 0;
> static lenSptype=0;
...
> static int instcount=0;
> instcount++;
> lenN += n.length();
> lenN2 += n2.length();
> lenSptype += sptype.length();
> if (instcount % 1000 == 1 ) {
> kdDebug() << "instcount (1) star objects == " << instcount << "
lenN=" << lenN << " lenN2="
> << lenN2 << " lenSptype=" << lenSptype;
The last output of this (when creating each star once) is
> stars: instcount (1) star objects == 125001 lenN=500737 lenN2=2903
lenSptype=250002
So the strings that we keep in the stars are small. I followed the
constructors and found nothing that uses 200 bytes.
At the moment I have no idea what is happening here.
Heiko
Heiko Evermann wrote:
> Hi everyone,
>
> I have analyzed the memory behavior of kstars:
> Our observation from the process stats is that we need about 500 bytes
> per star
> when we force loading of all stars.
>
> How big are the objects that we are dealing with?
> This can be found out by using sizeof:
> bool KStarsData::readStarData( void ) {
> kdDebug() << "size of QString " << sizeof (QString);
> kdDebug() << "size of Star Object " << sizeof (StarObject);
> kdDebug() << "size of dms " << sizeof (dms);
> kdDebug() << "size of QStringList " << sizeof (QStringList);
> kdDebug() << "size of double " << sizeof (double);
> kdDebug() << "size of bool " << sizeof (bool);
> kdDebug() << "size of SkyPoint " << sizeof (SkyPoint);
> kdDebug() << "size of SkyObject " << sizeof (SkyObject);
>
> The (commented) result is this:
>
> double:
> size=8 bytes
>
> bool:
> size=1 byte
>
> QString:
> size=4 bytes (for empty string, plus dynamically allocated content,
> which might be shared??)
>
> QStringList
> size = 4 bytes (for empty list, plus dynamically allocated content)
>
> dms:
> size=36 bytes (via sizeof)
> details:
> double D = 8 bytes
> mutable double Radians = 8 bytes
> mutable double Sin = 8 bytes
> mutable double Cos = 8 bytes
> mutable bool scDirty = 1 byte
> mutable bool rDirty = 1 byte
> sum: 34 bytes
> 2 bytes not accounted for. Rest might be padding to get a multiple of
> a dword as size
>
> SkyPoint:
> size=292 Bytes (via sizeof)
> dms RA0 = 36 bytes
> dms Dec0 = 36 bytes
> dms RA = 36 bytes
> dms Dec = 36 bytes
> dms Alt = 36 bytes
> dms Az = 36 bytes
> dms galLat = 36 bytes
> dms galLong = 36 bytes
> sum: 8*36 bytes = 288
> plus 4 bytes for the pointer to vtable
> all bytes accounted for
>
> SkyObject
> size=332 bytes (via sizeof)
> is a SkyPoint => 292 bytes, vtable pointer already included
> unsigned char Type 1 byte
> float Magnitude 4 bytes
> QString Name 4 bytes (+ variable content)
> QString Name2 4 bytes (+ variable content)
> QString LongName 4 bytes (+ variable content)
>
> QStringList ImageList 4 bytes
> QStringList ImageTitle 4 bytes
> QStringList InfoList 4 bytes
> QStringList InfoTitle 4 bytes
> QString userLog 4 bytes
> sum: 329 bytes accounted for. rest might be padding to get a multiple
> of dword as object size (compiler dependent?)
>
> StarObject
> size=364 bytes (via sizeof)
> is a skyObject => 332 bytes
> QString SpType => 4 bytes
> double PM_RA 8 bytes
> double PM_Dec 8 bytes
> double Parallax 8 bytes
> bool Multiplicity 1 byte
> bool Variability 1 byte
> sum: 362
> 2 bytes not accounted for, maybe padding to get a multiple of a dword
> as size
>
> This means that about 140 bytes are not accounted for. I am not sure
> whether they can
> be completely explained by string lengths of names. The names of most
> stars should be
> much simpler.
>
> A big byte consumer is dms, which is used several times. I know little
> to nothing about
> our algorithms. To me it makes sense to keep the catalog coordinates
> in double precision.
> However I am not so sure whether RA and Dec in skypoint (current true
> sky coordinates)
> need to be kept in double precision. I suppose that they are mainly
> used for projection
> to a 2d screen and are not used as a base for further calculations.
> Could it be enough
> to keep them in float precision by having two dms classes: one in
> double and one in float
> precision? This might save 36 bytes per star, which is about 10% of
> the memory load.
> And from a
>
> The two dirty flags (2 bits) make up for a total dword in memory
> consumption. This
> is unfortunate. But I see no easy way to get around this. One could
> set the sin/cos to
> illegal values (like >1), but then checking for dirty would result in
> a floating point operation,
> which is too expensive. Perhaps somebody else has an idea? We have 8
> dms objects in a
> StarObject. Each of wich spends 4 bytes on dirty flags, which is 32
> bytes of memory,
> or (again) about 10% of the memory load. Do all dms objects get
> invalid at the same time?
> Or do some (like the catalog coordinates) never get outdated?
> Do we need to keep D and radians in a dms object? Could the algorithms
> be rewritten so that
> we only ned one of them (without losing optimizations that we like a
> lot?)
>
> When I follow where data comes from, then I am tempted to say that the
> dms object is
> an expensive way of storing a double value (36 bytes instead of 8). It
> is quite a price to
> pay for optimization. Do we need sin/cos from all these values so
> frequently that we
> have to precompute them all? Just imagine that 8 bytes were enough:
> this would save
> 8*28 bytes per star, or 224 bytes, which is half the memory load of
> kstars.
>
> I think most star objects will only have one name? So we might
> introduce a class of its
> own for them? (saves 8 bytes for 2 QStrings, plus the string content,
> just in case
> QString does not recognize that one contains the same value as the
> other?)
>
> The whole area of QString memory management might be worth
> investigating. Maybe it is
> not as good as I think?
>
> And all this still leaves some bytes unaccounted for. Are there any
> memory leaks in the
> reading of the stars from disk? Files that are not closed etc? Has
> anybody checked for
> leaks recently?
>
> By the way: set both faint limits (zoomed in and out) to 1. Close and
> restart kstars.
> Go into settings, set limit to 8. The missing data does not get loaded
> in the current cvs...
>
> So much for tonight.
>
> Any comments, ideas etc?
>
> Greetings from Hamburg,
>
> Heiko
>
> _______________________________________________
> Kstars-devel mailing list
> Kstars-devel at kde.org
> http://mail.kde.org/mailman/listinfo/kstars-devel
>
>
More information about the Kstars-devel
mailing list