[Kstars-devel] strategies for expanding the size of catalogs

[Jason Harris]

Hi James,

On Monday 15 October 2007 16:01, James Bowlin wrote:
> I think we would get the best disk performance using flat binary
> files sorted by brightness as you suggest.  Unfortunately binary
> format will not be portable so we would probably want to generate
> the binary files at install time or have separate data sets for
> different architectures.  Also, I don't know if C++ is designed
> to let us initialize objects by just loading in memory.  In C
> and assembly, I am accustom to reading from disk directly into
> essentially an array of structs.
>
I hadn't considered the non-portability issue.  Probably creating the binary 
data files at compile time is the way to go.  Someone in KDE-land must be 
doing this already...

It would be nice if we knew ahead of time how much of the data file to read to 
reach magnitude X.  However, I think the variable-length string names make 
this difficult, unless we pad each object's name data to be the same number 
of characters.  Anyway, maybe this is not important, if we will just be 
reading records sequentially from the beginning of each file.

> Another idea would be to use a MySQL backend but I think we can
> actually get better performance out of flat binary files.
>
> Instead of (or in addition to) giving users a faintness magnitude
> slider we could give them a slider for how much memory to use for
> holding stars.  The idea is that we can usually keep all of the
> stars we display in RAM and only hit the disk when the user zooms
> or pans (or the time changes significantly).  We should avoid
> hitting the disk on every draw cycle if at all possible, maybe we
> should just refuse to do it except in very special situations (such
> as with automatic updating turned off).
>
Good idea.  We'll need some way to keep track of which stars are currently 
loaded into memory.  Maybe something as simple as a bool for each HTM trixel?

> A crude form of progressive drawing would be to first draw
> the visible stars we have in memory then hit the disks and
> trigger another draw after the new stars have been loaded.
> I tried doing this when the faint magnitude was increased but
> I had to give it up because the splash screen caused the main
> window to gray out so the first draw with just the stars already
> in memory was immediately erased.  We might want to solve this
> existing problem before enlarging the catalog.
>
> One potential show stopper would be proper motion.  We currently
> re-index parts (or all of) our catalog whenever the user makes
> a very large change in the time.  I don't know what to do about
> this.

Can we reindex in a separate thread to keep the UI responsive, and rely on the 
old index in the meantime?  For most stars, the index is not going to change 
even if they have moved significantly.  It's only those objects whose motion 
happens to carry them across a trixel boundary that we have to worry about.  
Also, we have the additional buffer that the area covered by the "visible" 
trixels extends beyond the rectangular boundary of the SkyMap widget.

Imagine the case where the display is zoomed in on some region of the sky, and 
the user makes a big timestep.  There is a star that was in a non-visible 
trixel with the old index, but after re-indexing it will be found to have 
moved into a visible trixel.  I don't think it's terrible if that star 
appears some seconds after the time change is made (as might happen if we 
reindex in a background thread).  Again, most of the stars drawn on screen 
won't be affected by the reindexing, so the user probably won't even notice 
the reindexing.

This is a classic "corner case"; it's kind of frustrating to spend so much 
effort on something that no one will notice :)  But kind of satisfying to 
come up with the right way to do it.

regards,
Jason

-- 
Jason Harris
jharris at 30doradus.org