[Marble-devel] Re: Natural Earth / Vector Files [Was: Marble Sprint]

[John Layt]

On Sunday 10 October 2010 07:54:05 Torsten Rahn wrote:
> Am Sonntag, 10. Oktober 2010 02:08:48 schrieb John Layt:
> > The main benefit for the second case is the embedded data attributes such
> > as country code, groupings, relative magnitude, etc, which will allow
> > easy linking and manipulation.  We could do something similar using PNT,
> > but we would have to create our own attribute storage to link things
> > together, and then maintain the attribute data, so sticking with the
> > shapefiles would seem a better solution there.

Thinking about it some more, there seems little point in having the Natural 
Earth data available in 2 different formats, and forcing anyone wanting a 
stats view to download it in the shapefiles, taking up twice the disk space 
too.  It seems such a core function that making people jump through hoops is 
counter-productive. We will already have the data, we should use it.  

So that implies we need to extract the Natural Earth metadata from the 
shapefile as we convert it into PNT, and store that metadata somewhere and use 
it for connecting everything.  If we use an even more efficient format like 
you suggest below and do the detail level on the fly, and accept an increase 
in the data size we'll get both for the price of one.

Shapefile import would be left for another time.


> We could change this to something like this:
> 
> 1st integer (32 bit): Latitude in arcseconds + highest bit indicates new
> polygon starts: header information has to be read from 3rd integer
> 2nd integer (32 bit): Longitude in arcseconds
> optional 3rd integer: would store Geonames.org ID of the feature +  highest
> bit indicates whether it's a linestring or linearring.

OK, so take the entire 1:10 dataset, convert it to PNT2 format, each point 
takes 64 bits/8 bytes, except the start of polygon which takes 96 bits.  So 
roughly 533,202 x 8 bytes = 4 Mb for the country borders alone (I'll do full 
calculations later).  If that's too much to ship, then ship the 1:50 dataset 
(with its coarser detail than the current data) and download the 1:10 asap.

Rather than the Geonames ID, we could just use the Natural Earth object ID, 
then a look-up file/table that matches the NE ID to the ISO / FIPS / whatever 
code (NE provides this in the metadata) and Geonames ID (which we would have 
to provide).  This would allow look-ups via whatever code or ID is available, 
and we wouldn't be reliant on Geonames IDs staying constant.

So required work is:
1) New PNT file format definition (with a different name, MBL?)
2) Metadata file format definition (could just be csv?)
3) shp2mbl script to convert shp to new formats (using Perl::shp? there's 
shp2xxx scripts out there we could copy?), including matching to Geonames ID
4) New MBL file loading code.

> 2.) Via GeoPainter and GeoDataLineString ("libgeodata"): This is the new
> "clean" approach. But it has got a feature regression: LinearRings (=closed
> polygons) which contain one of the poles inside don't get  rendered
> properly yet (Think of Antarctica).
> We are using this to render KML and GPX already. Thanks to Thibault Gridel
> it can also render PNT, but we aren't using it yet due to the feature
> regression. Shape file support should be based on this approach.
> 
> Ideally we'd get rid of 1.) altogether really soon ;-)

I see little point in trying for a solution with (1) and then switching, so 
let's concentrate on (2), even if Antarctica gets a workaround in the interim.  
Leave (1) loading the base PNT files for now until everything in (2) works 
well enough to take its place.

So the required improvements there are:
1) Fix polar issue
2) Implement D-P reduction on the fly (assuming its fast enough!)


Thinking about the simple case of a Country Picker or Timezone Picker widget 
in kdelibs or kdebase, perhaps marble is too heavy a solution there.  Perhaps 
take the NE 1:110m shapefile, convert it into an SVG with data attributes and 
colouring and load that into a QGraphicsView.  Then when someone clicks on a 
shape, it returns the attribute for the country code or timezone name.  Keep 
it simple and lightweight.

Cheers!

John.