[Kst] NetCDF Data Compression (Packing) via signed 16-Bit-Integers (type SHORT INT)

[Rix, Patrick]

Dear KST-Users & Developers,

I'm a newbie to KST-plot and I've got a question concerning the interface for reading data in binary NetCDF format.
I found KST when searching for a tool for plotting time series result data of MBDyn<http://www.aero.polimi.it/mbdyn/>-multibody-simulations of our wind turbines.
MBDyn can use either ascii text format or binary NetCDF format (i.e. float numbers) for its output files.
I found that plotting of scalar (1D) time series of data type float (4 bytes) works well in KST, so I was wondering if also support for packed / compressed data could be added (if not already implemented) to the NetCDF interface as indicated by [Eq.1] below  ?

When simulating wind turbines we usually have to deal with a huge amount of data from a lot of simulations arising from parameter variations.
Fortunately there is a way to pack the data by using a simple transformation into 2-byte integer numbers of data type 'signed short int' which results in a small loss of information (i.e. precision) but 65.535 different niveaus of the signed short int type (ranging from -32.768 to 32.767) are usually sufficient for capturing the data range of a simulated float time series while the packed data only require half of the disk space than the (4 bytes) floating point values.

I found that this sort of simple packing algorithm seems to be part of the "commonly used" NetCDF conventions and being integrated in some NetCDF tools:
If a (time dependent, scalar 1D-) variable owns the special attributes  'scale_factor'   and   'add_offset'  (refer to: http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Attribute-Conventions.html) these indicate that the variable has been packed to short int format.
The original float values  'X(ti)'  of a time series at time index 'ti' can be retrieved back by applying its  'scale_factor'   and   'add_offset'  attributes (saved in the NetCDF file for each variable) to the short integers  'I_short(ti)'  :

[Eq.1]     X(ti)  =  I_short(ti)  *  scale_factor   +  add_offset

The values for  'scale_factor'   and   'add_offset'   (which are floating point numbers) were derived during the packing procedure by scanning the float time series  'X(ti)'  for its ultimate values  'Xmax' and  'Xmin'  marking the data range of the time series   Xrange = Xmax - Xmin

  scale_factor  =  Xrange  /  DataTypeResolution_short

  add_offset  =  Xmin  +  0.5 * Xrange

with  'DataTypeResolution_short'  being the resolution of data type short int, e.g.

   DataTypeResolution_short  =  65.000

The integer numbers  I_short(ti)  were calculated by

   I_short(ti)  =  round(  ( X(ti) - add_offset ) / scale_factor  )



è     Does anybody know if something similar has been already implemented into KST-plot
           orr
è      (if not) what would be the best and easiest way to implement the backtransformation according to [Eq.1]  ?

Any help on that will be warmly appreciated.

With kind regards

Patrick Rix
Loads & System Simulation

REpower Systems SE
TechCenter
Albert-Betz-Str. 1
D-24783 Osterrönfeld

Tel.: +49 - (0) - 4331 - 1313 - 9408
Fax: +49 - (0) - 4331 - 1313 - 9954

e-mail: p.rix at repower.de<mailto:p.rix at repower.de>
Internet: www.repower.de<http://www.repower.de>



________________________________
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte umgehend den Absender und löschen Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist nicht gestattet. Wie Sie wissen, kann die Sicherheit von Übermittlungen per E-Mail nicht gewährleistet werden, E-Mails können missbräuchlich unter fremdem Namen erstellt oder verändert werden. Aus diesem Grund bitten wir um Verständnis dafür, dass wir zu Ihrem und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden Erklärungen ausschließen müssen. Diese Regelung gilt nur dann nicht, wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung über die Einhaltung von Sicherheits- und Verschlüsselungsstandards getroffen haben.

This e-mail contains confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. As you know, the security of e-mail transmissions can not be guaranteed. E-mails can be misused to be written or modified under false names. For that reason, we ask you to understand the necessity for us to rule out the legal obligation of the above statement, for your protection and ours. This regulation is only invalid if we have concluded a special written agreement with you about the compliance with security and encryption standards.

REpower Systems SE · Sitz: Hamburg · Vorstand: Andreas Nauen (Vorsitz), Matthias Schubert, Marcus A. Wassenberg, Vinod R. Tanti · Aufsichtsratsvorsitzender: Tulsi Tanti · Registergericht: AG Hamburg (Mitte) HRB Nr.: 118644
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kde.org/pipermail/kst/attachments/20130228/598b9ac2/attachment.html>