[Kstars-devel] strategies for expanding the size of catalogs

[James Bowlin]

On Mon October 15 2007, Jason Harris wrote:

> I hadn't considered the non-portability issue.  Probably creating the
> binary data files at compile time is the way to go.  Someone in
> KDE-land must be doing this already...

I usually create data files like this at runtime with a special command
and use a write() function that is analogous to the read() we are going
to use to read the data back in.  In fact I'm doing something like that 
already with cbounds-?.idx files.  I have it set up so you can 
uncomment one line (or something) and then the data for those files 
gets spit out.  The tricky part might be to convince C++ to play nice 
with the star data when we read it back in as a simple read to memory,
but maybe it will just work.

> It would be nice if we knew ahead of time how much of the data file
> to read to reach magnitude X.  However, I think the variable-length
> string names make this difficult, unless we pad each object's name
> data to be the same number of characters.  Anyway, maybe this is not
> important, if we will just be reading records sequentially from the
> beginning of each file.

I don't think any of the dimmer stars will have names so fixed fields
should not be a problem.  We can easily create indexes for each file
telling us how far down certain magnitudes are.  I've already done
something like this in the file starlnum.idx.  It contains the line
number of stars with 92 different magnitudes.  I use this for giving
reasonable percentages when loading the star data.  You can see that
it always ends at (or near) 100% even if you're not loading all the
stars.


> Good idea.  We'll need some way to keep track of which stars are
> currently loaded into memory.  Maybe something as simple as a bool
> for each HTM trixel?

Yes, or it could be a null pointer.  Anything on a per trixel basis 
should be fine.


> > One potential show stopper would be proper motion.  [...]
>
> Can we reindex in a separate thread to keep the UI responsive, and
> rely on the old index in the meantime?  For most stars, the index is
> not going to change even if they have moved significantly.  It's only
> those objects whose motion happens to carry them across a trixel
> boundary that we have to worry about. Also, we have the additional
> buffer that the area covered by the "visible" trixels extends beyond
> the rectangular boundary of the SkyMap widget.

The problem is that if a star is in the wrong trixel then it won't be 
loaded into memory at all so there is no way for us to apply the 
correction to move it into the correct trixel.  I don't want to have to 
reindex every star on disk.  The only solution I can think of ATM is to 
pre-calculate the index for various epochs and then keep lists of the 
deltas (the stars that change trixels).  Extra copies of these stars 
then might be kept in separate files. This is starting to get messy 
because we will end up with thousands of files.  For each trixel we
we have the main data file and then files with the deltas for various 
epochs.

I just don't know of a good way to do this.


> I don't think it's terrible if that star appears some seconds after
> the time change is made (as might happen if we reindex in a background
> thread).  

Unfortunately we can't re-index all the stars in just a few seconds.
Just reading all the data files will take a very long time.

> This is a classic "corner case"; it's kind of frustrating to spend so
> much effort on something that no one will notice :)  But kind of
> satisfying to come up with the right way to do it.

I agree it is a tiny corner case.  But the problem we face now is
that if we don't do something clever, more and more stars will simply 
disappear as the time moves away from J2000.


-- 
Peace, James