very large data files

[Robert Knight]

Hi Jason,

I haven't looked at how you store the data but will you run into
performance problems loading such a large data file into memory in any
case?

Does anyone provide a streaming service which allows you to obtain only
the subset of the data which is of interest? - Like Google Earth but for
stars.
 
Regards,
Robert Knight.

On Wed, 2008-07-09 at 21:25 -0700, Jason Harris wrote:
> Hello,
> 
> Our GSoC student is working on making KStars display 1 million stars,
> without adversely affecting the program's responsiveness.  He's doing
> so well that we're already making plans to try 10 million, 100
> million, and even (when we're feeling insane) 1 billion stars.
> 
> The problem is that the binary data file for even 1 million stars is
> already 74 MB.  This is larger than the entire kstars directory in
> svn, so it's clearly not something we want to store in the kdeedu
> module.  I'm wondering if it's even too big to make available through
> GHNS (although I guess it would be okay as long as we note the large
> file size in the description text).
> 
> You can guess what the real problem is: the 10 million star data file
> is going to be about 750 MB, and 100 million stars will be 7.5 GB.
> How are we going to distribute these files?  Is KDE billed per-bit for
> transfers from kde.org?
> 
> You'll note I didn't even mention the file containing 1e9 stars.  The
> US Naval Observatory publishes a catalog with 1e9 stars called NOMAD
> (which our large catalogs are based on), and to obtain it you have to
> send them a USB hard drive with a capacity of at least 100 GB.  We'd
> probably have to use a similar method.
> 
> thanks for any advice,
> Jason