[Kstars-devel] Additional Catalogs

[Akarsh Simha]

Hi Victor

My apologies, I've been rather busy over the week, and needed the
weekend to find time to look through your proposal.

At first glance, it looks very good, but I needed to discuss a couple
of issues. It looks like those are the very same issues you have
raised.

> The first step would be on defining a conversion tool from the catalog
> native format to the one decided by us (actually an abstract class
> extended by other particular classes overriding a read method for each
> catalog). This will ensure that if any particular catalog would be
> added later, then simply overriding a read method and executing a
> parse method would add the new catalog to the implemented structure.
>
> The tough part it's the actual way of storing the data and efficiently
> access it so that it can be displayed on the user's screen.
> Some questions/suggestions:
> 1) Split/sort the stars in the catalog in sky data file chunks, in
> directories by their positions (ra, dec) on the sky. As the zoom
> factor increases, stars are dynamically loaded from the according file
> chunks.
>
> 2) What exactly the database handling prerequisite means? I'm used
> working with relational database systems, but I was wondering how
> exactly it's related to the final result of the project. It is about
> using it as an auxiliary tool to generate the final binary files for
> catalogs (as I have seen done in the KStars code for the current
> catalogs) or is it an alternate way of storing the catalogs?
> I've documented also about non relational database management systems
> (MongoDB [1], particularly) as I've seen many people reporting
> increased performance. But, whatever the database used would be, I'm
> (currently) not seeing a small database server running on localhost in
> paralel with kstars (even though this might actually solve a lot of
> problems) and I guess that certainly it's _not_ an option to store
> catalogs and load star information from a remote server.

Okay. Very valid points. Let me first jot down the ideal requirements
that we (or at least I) would like to have in KStars. Please do pitch
in if you have alternate ideas / suggestions:
1. Reasonably fast load / unload of data.
2. Easy filtering by magnitude, location in the sky, HTM trixel (I
suppose you've already read about this) etc
3. Easy retrieval of data required.
4. Alternate designations for the same object -- a search for any
designation should yield the object.

Ideally, I felt that an SQL database would easily meet the
requirements. While it will definitely not suit the requirements of
star catalogs, which are extremely large, it would work for deep-sky
catalogs. We intend to add many more deep-sky catalogs in the future,
because any reasonably experienced amateur astronomer would find that
NGC and IC catalogs are not enough. Besides, alternate designations
are a pain. M 17, Omega Nebula, Swan Nebula, NGC 6618 all mean the
same thing!

Clearly, we shouldn't do this for stars. But what's the advantage for
deep-sky? An immediate consequence is you could make very custom
searches over the database using SQL.

Note that star designations are also a problem -- Ideally, we should
be able to search by HD, SAO, GSC and HIP catalog numbers. We already
handle HD catalog numbers in KStars, but handling other catalog
identifications might require us to organize the binary data files for
USNO NOMAD differently. So, I see this as a difficult problem.

Relegating the star catalog problem for later, I feel that the first
step would be to improve the DSO catalog handling.

Suppose we decide to use an SQL database, then there is a standalone
SQL database called SQLite. It doesn't require you to run a server at
all. I haven't used it before, only heard of it -- but I think that
should serve our purpose.

Do let me know what you think of these issues.

> 3) I've read a lot about STXXL [2] in the Boost Library and I think
> this would be a really great improvement in the way KStars works with
> very large binary files in which the catalogs would be stored. It has
> a great variety of options that could be really useful if KStars will
> have to deal with a lot of catalogs, and I'm suggesting on using it in
> the project.
> For example, limiting the memory used, very efficient I/O operation
> implementation in their base layer, working with external memory
> (right what extra catalog support means), etc.

That's a very interesting library! I would like to understand how you
plan to fit this into our HTM model. (I had a brief look at the site,
and this isn't very clear to me). As you probably know, we divide the
sky into trixels, and each HTM trixel has a list of stars associated
with it, which is read to the required faintness. Blocks of stars that
have been read into memory are cached to avoid unnecessary overhead
while panning. I'd like to understand how this would translate as
STXXL containers.

Regards
Akarsh