[Marble-devel] server space for marble data

[Jens-Michael Hoffmann]

Hi,

Am Mittwoch, 3. Oktober 2012, 17:21:14 schrieb Tom Albers:
> Hi,
> 
> We can not host 2TB on any of our servers currently. That's just an
> enormous amount of data. We can rent another server from Hetzner, this
> costs around 650 euro per year[1]. If you can provide the funds for that,
> we are happy to set it up and maintain it.
> 
> Please note that with this amount we can not utilise our KDE mirror
> network. That means the server should be able to manage on it's own. It's
> quite a difference if it is one 2TB file or a 1 million files of 2KB. Also
> the amount of concurrent users should be factored in. From Hetzner we get
> a 10TB limit a month. So if this is a single 2TB file, we can not serve
> that much people...
> 
> In short: this requires a bit more thought and input from you guys. As you
> are running this privately, you should be able to provide those details.
> Also I think you can, and should, start a fundraiser to get the funds for
> this (or discuss options with the board).

I got some numbers from a openstreetmap tile server admin yesterday. 

On October 2nd, Marble users caused
- 111611 uncached tile requests from
- 503 unique ip addresses
- top 50 ip's were requesting between 14858 and 461 tiles
- around 9kb each tile on average
- cache hit ration is between 60% and 85%

That would mean - assuming 85% cache hits - 744073 tiles/day (a 9kb) served => 
6.4GB/day tiles.

When interpreting these numbers the following should be taken into account:

1) expiration time for osm tiles is 7 days, we can use a higher value as we 
don't expect our data to change very often, there may be improvements over 
time, but I'd say 1 month expiration time or even more should be ok. Using a 
higher value here will cause Marble to "refresh" less often, won't help with 
"initial" requests

2) file size stats for a subset of around 440000 of our tiles:
size range[kb]   percentage[%]
2-4               2.4
4-8               2.7
8-16              4.8
16-32             4.6
32-64            15.2
64-128           27.3
128-256          43.0

If you need more information or something is unclear, please do not hesitate 
to ask.


kind regards,
Jens-Michael


> 
> Best,
> 
> Tom Albers
> KDE Sysadmin
> [1] Based on EX4 from
> http://www.hetzner.de/hosting/produktmatrix/rootserver-produktmatrix-ex
> 
> ----- Oorspronkelijk bericht -----
> 
> > Hi,
> > 
> > On 03.10.2012 16:34, Jens-Michael Hoffmann wrote:
> > > The routing data (atm hosted by Dennis Nienhüser privately) also
> > > requires some
> > > additional server space.
> > 
> > just a minor correction, the routing data is hosted on filesmaster
> > already, just the conversion routine (which runs 24/7 to incorporate
> > recent changes in OpenStreetMap data) is running on my private
> > server.
> > We could use additional space though for more variants, and possibly
> > vector OpenStreetMap data.
> > 
> > Regards,
> > Dennis