question about kstars catalogs

[Vincent Hourdin]

Thank you so much Akarsh for your very detailed message and Jasem for
the link. It took me a while but I finally was able to use the four
catalogs in Siril for plate solving (siril has its own) and photometric
color calibration.
I copied the htmesh library in siril source tree and wrote a binary
writer from the examples you provided and the C sample tools that were
in KStars source tree for catalog import/transformation/checking.
It's not yet merged, but if you want to have a look it's there:
https://gitlab.com/free-astro/siril/-/tree/829_local_nomad/subprojects/htmesh
https://gitlab.com/free-astro/siril/-/tree/829_local_nomad/src/io/kstars
https://gitlab.com/free-astro/siril/-/blob/829_local_nomad/src/io/catalogues.c

I don't need to modify the catalogs, they fit our needs as is, thank you.
I especially needed the B and V information, or B-V, so your NOMAD
catalog, limited to mag 18 proved perfect as it is compact and contains
all the stars with this photometric information.

I noticed a few incoherencies in the binary file headers, in case you
didn't already know:
- the 3 first catalogues have a magnitude multiplier of 100
  while it's 1000 for nomad, but nomad's header reports it as 100 too
- nomad's scale for dRA and dDec is also incorrect, reported to 10, but
  it's already in mas/yr
- for namedstars and unnamedstars, the scale of dRA and dDec is 10 as
  reported, but for deepstars (Tycho-2), the data seems completely
  random compared to what vizier gives for the few stars I've looked up.
  I don't know the scale but it should simply not be used if my sample
  was not just unlucky. Anyway, from what I could see, the proper motion
  for Siril's use of near current time is basically insignificant.

Next step for Siril on that side would be to document the use of the
catalogs and also distribute them. It was not easy to find the
deepstars.dat file, and the nomad file can be compressed to 1 GB with xz
instead of 1.4 GB with the current gzip. I was thinking I would put
online the 4 catalog files to have them easily available for Siril
install, with bittorrent from a few computers/servers. One question I
have is about the license, I don't know if I am allowed to do that, I
don't know what are the terms of use of the catalogs either, can you
enlighten me on this subject?

I haven't yet looked at the object names databases, it's not as
important for now, but at least I know where to look if needed, thank
you!

Vincent

On Wed, Jun 08, 2022 at 10:34:50AM +0300, Jasem Mutlaq wrote:
> Thank you for the detailed response Akarsh. We documented stars handling in
> the API here: https://api.kde.org/kstars/html/Stars.html
> 
> --
> Best Regards,
> Jasem Mutlaq
> 
> 
> 
> On Wed, Jun 8, 2022 at 9:28 AM Akarsh Simha <akarshsimha at gmail.com> wrote:
> 
> > Hi Vincent
> >
> > Thank you for your contributions to Siril, it's an excellent piece of
> > software and fills a crucial need in the open-source software ecosystems.
> >
> >>
> >
> >
> > Hello, I'm a developer of Siril, I'm looking at options to integrate a
> >> photometric star catalog for offline operation ; we currently use NOMAD
> >> and APASS from the vizier web service.
> >
> >
> > I'm not sure exactly what your use-case here is, but please be warned that
> > NOMAD was generated through algorithms running on images without human
> > curation, so at least in the version used in KStars there are several
> > artifacts, especially around milky way patches, diffraction spikes of
> > bright stars, and cores of galaxies. Perhaps there is a better version
> > released, but I haven't kept up.
> >
> >
> >> I'm looking for more information about the kstars catalog binary format,
> >> I was hoping you could give me pointers.
> >
> >
> > The KStars star catalog binary format is documented here:
> >
> > https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/stars.dox
> > (for some reason I can't find the doxygen-generated HTML on the web)
> >
> > I believe that the same file also has details of how the Hierarchical
> > Triangular Mesh index is structured. If you search "Hierarchical Triangular
> > Mesh" on the internet, you will find some papers, but a useful page is on
> > the Sloan DSS survey:
> > http://www.skyserver.org/htm/
> >
> > These days HEALPix is more popular. It has more complicated mesh
> > structure, but they tend to prefer it due to the avoidance of
> > compute-intensive trigonomeric functions. I believe KStars implements both
> > HTMesh (for stars and deep-sky objects) and HEALPix (for progressive
> > surveys). The HTMesh implementation in KStars, as a C++ library, lies here:
> > https://invent.kde.org/education/kstars/-/tree/master/kstars/htmesh
> > I believe the above is a standalone library. There is also a Perl wrapper
> > around it:
> >
> > https://invent.kde.org/education/kstars/-/tree/master/kstars/data/tools/HTMesh-0.01
> >
> > When we built this system back in 2008, the approach we took was an
> > optimization-first approach. This resulted in many non-portable decisions,
> > like using a custom-designed binary file format, restricting to 16- and
> > 32-byte data structures, and C-style memory allocations. If I were to
> > redesign this system today, the first thing I'd try is to use an existing
> > low-footprint, fast, serverless-database. The binary file is essentially a
> > key-value database indexed by trixel as the key, mapping to an array that
> > has entries sorted by magnitude within each trixel. A separate index file
> > is provided for lookup by Henry-Draper catalog number.
> >
> > There are several other subtleties in the system, including replicating
> > some high-proper motion stars in multiple trixels that they may appear in
> > over the course of 10000 years. Most of these are explained in the doxygen
> > file.
> >
> >>
> > I believe I have the raw text data from the USNO on my hard drive, if it
> > is easier to start from there rather than from the binary file. But
> > obviously, this does not have a HTM index associated with it.
> >
> > The code in kstars/data/tools unfortunately does a weird rigmarole of
> > loading this data into a MySQL database, doing some intermediate processing
> > there, and then writing out the binary file from the MySQL. At that time,
> > it made sense to load the data into MySQL, but today, SQLite would probably
> > be a better choice.
> >
> > Unfortunately, this has made debugging or extending the catalog hard.
> > Please be aware that we have some issues reported on our Gitlab, I believe
> > they mostly concern Tycho2-stars.  Hipparcos and Tycho-2 (I believe?)
> > stars have been removed from the NOMAD catalog binary file supplied in
> > KStars so as to avoid duplication, and they are shipped in separate binary
> > files (There's four files in total: namedstars.dat, unnamedstars.dat,
> > deepstars.dat and the NOMAD-1e8.dat -- the first two I believe are
> > Hipparcos, third Tycho-2, and last USNO NOMAD, although I could be wrong).
> >
> > I am going to propose that if you are going to take the effort to re-work
> > this pipeline, we might as well collaborate and do it in such a manner that
> > would be beneficial to both projects. This way, our users need to install
> > the catalog only once. I am very open to porting away from the binary file
> > format, although I may not be able to commit the time to do the work today.
> >
> > I've been looking at the code and binary format as shown in
> >> nomadbinfiletester.c, I think it would not be difficult to use this to
> >> create a simple search of stars in NOMAD, not using a database but
> >> simply the file organized in "trixels".
> >
> >
> > Yes, this should be possible. In fact our binary file format already has
> > this solution. The header contains the 64-bit offsets into the file that
> > correspond to each trixel, and at each of those offsets, one finds a small
> > header followed by the stars organized by magnitude, brightest first.
> >
> >>
> >>
> >> I didn't find how the trixels were distributed in the sky, is there a
> >> code you can point me to that converts RA,DEC to a trixel number?
> >> In Siril we only need to get a list of stars within a radius or a
> >> square around a target's coordinates, with a filter on magnitude too.
> >
> >
> > See the SDSS documentation for a visualization of how trixels are
> > distributed in the sky:
> > http://www.skyserver.org/htm/
> > An important parameter is the "level" of the HTM, which is the number of
> > nested levels up to some constant. Each binary file catalog specifies the
> > level of HTM that it's indexed on in the header.
> >
> > The trixel mesh operates, as you can expect, in J2000 coordinates.
> >
> > As for finding a trixel corresponding to an (RA, Dec): you are looking for
> > HTMesh::index
> >
> > https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L72
> >
> > For the latter two usecases, finding the trixels covered by a region, you
> > are looking for the various overloads of HTMesh::intersect:
> >
> > https://invent.kde.org/education/kstars/-/blob/master/kstars/htmesh/HTMesh.cpp#L104
> > I believe you can then iterate over the returned trixels through some
> > iterator structure.
> >
> > You may also find the wrapper in SkyMesh.cpp worth looking at, although it
> > has several KStars-specific patterns:
> >
> > https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/skymesh.cpp
> >
> > For an example use of the mesh, see:
> >
> > https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/deepstarcomponent.cpp#L283
> > (I believe aperture is a wrapper around intersect)
> >
> > The intersect call is on line 283, which is followed by getting the
> > iterator on the next line, and the iteration loop is seen on line 310.
> >
> > The deep-sky object implementation also uses HTMesh, but the data is
> > instead stored in a SQLite database:
> >
> > https://invent.kde.org/education/kstars/-/blob/master/kstars/skycomponents/catalogscomponent.cpp#L92
> >
> >>
> >
> >> Another thing I was wondering is how to convert object names to
> >> coordinates (coordinates to name could be useful too), I guess you have
> >> a list in kstars, but would it easy to extract from whichever catalog
> >> contains them?
> >>
> >
> > In KStars, each class of objects is managed differently when it comes to
> > name.
> >
> > I believe all stars that have names are always held in memory and are
> > never loaded. For this, there is a supplemental binary file called
> > starnames.dat that contains the names of the stars. Each star's data in the
> > binary file may contain some catalog numbers (notably Henry-Draper). The
> > reverse mapping is stored and distributed as Henry-Draper.idx (apparently,
> > this is broken right now; I filed a number of issues on invent.kde.org
> > related to this recently).
> >
> > For deep-sky objects, we store them in the SQLite database. The object
> > names are dynamically queried against the database via the
> > CatalogsComponent::findByName() wrapper, and the results are loaded into
> > memory (not sure of the details) perhaps upon centering the object, simply
> > because we load the trixel's contents.
> >
> > For all other objects (eg: planets / comets ...), the corresponding
> > subclass of SkyComponent supplies the names (look for something like
> > objectNames).
> >
> > Finally, if a name is not known to KStars, we submit a query to CDS Sesame
> > (which is an API front-end to SIMBAD and VizieR, and ostensibly NED
> > although it's never worked for me), parse the result, store it in the
> > SQLite deep-sky object database and then render it on screen for the user.
> >
> > So you could pick our catalogs workflow created by Valentin Boettcher to
> > generate SQLite tables with information from VizieR:
> > https://protagon.space/catalogs/
> > I am unable to locate the repo which contains the code, and I'm not even
> > sure if it is publicly accessible. Perhaps Jasem / Valentin can comment on
> > that.
> >
> > Depending on how you intend to proceed, we may be interested in finding
> > ways to modernize our implementation of star catalogs in KStars by
> > leveraging any portable implementations you plan to make.
> >
> > Regards
> > Akarsh
> >
> >
> >
> >>
> >> Thanks a lot for your work and your help!
> >>
> >> Vincent
> >>
> >