very large data files
robertknight at gmail.com
Thu Jul 10 06:03:58 BST 2008
I haven't looked at how you store the data but will you run into
performance problems loading such a large data file into memory in any
Does anyone provide a streaming service which allows you to obtain only
the subset of the data which is of interest? - Like Google Earth but for
On Wed, 2008-07-09 at 21:25 -0700, Jason Harris wrote:
> Our GSoC student is working on making KStars display 1 million stars,
> without adversely affecting the program's responsiveness. He's doing
> so well that we're already making plans to try 10 million, 100
> million, and even (when we're feeling insane) 1 billion stars.
> The problem is that the binary data file for even 1 million stars is
> already 74 MB. This is larger than the entire kstars directory in
> svn, so it's clearly not something we want to store in the kdeedu
> module. I'm wondering if it's even too big to make available through
> GHNS (although I guess it would be okay as long as we note the large
> file size in the description text).
> You can guess what the real problem is: the 10 million star data file
> is going to be about 750 MB, and 100 million stars will be 7.5 GB.
> How are we going to distribute these files? Is KDE billed per-bit for
> transfers from kde.org?
> You'll note I didn't even mention the file containing 1e9 stars. The
> US Naval Observatory publishes a catalog with 1e9 stars called NOMAD
> (which our large catalogs are based on), and to obtain it you have to
> send them a USB hard drive with a capacity of at least 100 GB. We'd
> probably have to use a similar method.
> thanks for any advice,
More information about the kde-core-devel