[Kst] NetCDF Data Compression (Packing) via signed 16-Bit-Integers (type SHORT INT)

Sun Mar 3 22:01:44 UTC 2013

Hi Patrick,

Thanks for your message. I hope you enjoy using Kst with netCDF, which is y pretty nice format. 
I have implemented the netCDF datasource so I can tell you: the convention is not taken into account as of now. It should be fairly easy to add, but could you provide a sample file so that I can cross-check the code once implemented? Also, I would need to understand how to retrieve all the variable names. It is not clear to me from your mail whether each 2-byte int variable contains one data series, or if you have e.g. two data-series packed into a 4-byte int? 
I think the change should be pretty trivial to implement and if it's a clearly documented convention, we could probably apply it unconditionally, the other option being to add a configuration dialog, as the ASCII source has.
If you know a bit of C, I could point you to the right places in the code so that you try your luck at making the change. You can then provide me a patch that I'll integrate. Otherwise I'll do it, but theses days I'm unfortunately not as responsive as I'd like since I have very challenging deadlines in my work and a busy personal life. However, this change sounds small enough that I should be able to do it in a reasonable time.

Best regards,

Nicolas 

----- Mail original -----
> De: "Patrick Rix" <Patrick.Rix at repower.de>
> À: kst at kde.org
> Envoyé: Jeudi 28 Février 2013 11:14:07
> Objet: [Kst] NetCDF Data Compression (Packing) via signed 16-Bit-Integers (type SHORT INT)
> 
> 
> 
> 
> 
> Dear KST-Users & Developers,
> 
> 
> 
> I’m a newbie to KST-plot and I’ve got a question concerning the
> interface for reading data in binary NetCDF format.
> 
> I found KST when searching for a tool for plotting time series result
> data of MBDyn -multibody-simulations of our wind turbines.
> 
> MBDyn can use either ascii text format or binary NetCDF format (i.e.
> float numbers) for its output files.
> 
> I found that plotting of scalar (1D) time series of data type float
> (4 bytes) works well in KST, so I was wondering if also support for
> packed / compressed data could be added (if not already implemented)
> to the NetCDF interface as indicated by [Eq.1] below ?
> 
> 
> 
> When simulating wind turbines we usually have to deal with a huge
> amount of data from a lot of simulations arising from parameter
> variations.
> 
> Fortunately there is a way to pack the data by using a simple
> transformation into 2-byte integer numbers of data type ‘signed
> short int’ which results in a small loss of information (i.e.
> precision) but 65.535 different niveaus of the signed short int type
> (ranging from -32.768 to 32.767) are usually sufficient for
> capturing the data range of a simulated float time series while the
> packed data only require half of the disk space than the (4 bytes)
> floating point values.
> 
> 
> 
> I found that this sort of simple packing algorithm seems to be part
> of the “commonly used” NetCDF conventions and being integrated in
> some NetCDF tools:
> 
> If a (time dependent, scalar 1D-) variable owns the special
> attributes ‘scale_factor’ and ‘add_offset’ (refer to:
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf/Attribute-Conventions.html
> ) these indicate that the variable has been packed to short int
> format.
> 
> The original float values ‘X(ti)’ of a time series at time index ‘ti’
> can be retrieved back by applying its ‘scale_factor’ and
> ‘add_offset’ attributes (saved in the NetCDF file for each variable)
> to the short integers ‘I_short(ti)’ :
> 
> 
> 
> [Eq.1] X(ti) = I_short(ti) * scale_factor + add_offset
> 
> 
> 
> The values for ‘scale_factor’ and ‘add_offset’ (which are floating
> point numbers) were derived during the packing procedure by scanning
> the float time series ‘X(ti)’ for its ultimate values ‘Xmax’ and
> ‘Xmin’ marking the data range of the time series Xrange = Xmax –
> Xmin
> 
> 
> 
> scale_factor = Xrange / DataTypeResolution_short
> 
> 
> 
> add_offset = Xmin + 0.5 * Xrange
> 
> 
> 
> with ‘DataTypeResolution_short’ being the resolution of data type
> short int, e.g.
> 
> 
> 
> DataTypeResolution_short = 65.000
> 
> 
> 
> The integer numbers I_short(ti) were calculated by
> 
> 
> 
> I_short(ti) = round( ( X(ti) – add_offset ) / scale_factor )
> 
> 
> 
> 
> 
> 
> 
> è Does anybody know if something similar has been already implemented
> into KST-plot
> 
> orr
> 
> è (if not) what would be the best and easiest way to implement the
> backtransformation according to [Eq.1] ?
> 
> 
> 
> Any help on that will be warmly appreciated.
> 
> 
> 
> With kind regards
> 
> Patrick Rix
> Loads & System Simulation
> 
> 
> REpower Systems SE
> TechCenter
> Albert-Betz-Str. 1
> D - 24783 Osterrönfeld
> 
> 
> Tel.: +49 - (0) - 4331 - 1313 - 9408
> Fax: +49 - (0) - 4331 - 1313 - 9954
> 
> 
> e-mail: p.rix at repower.de
> Internet: www.repower.de
> 
> 
> 
> 
> 
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
> E-Mail irrtümlich erhalten haben, informieren Sie bitte umgehend den
> Absender und löschen Sie diese E-Mail. Das unerlaubte Kopieren sowie
> die unbefugte Weitergabe der in dieser E-Mail enthaltenen Daten ist
> nicht gestattet. Wie Sie wissen, kann die Sicherheit von
> Übermittlungen per E-Mail nicht gewährleistet werden, E-Mails können
> missbräuchlich unter fremdem Namen erstellt oder verändert werden.
> Aus diesem Grund bitten wir um Verständnis dafür, dass wir zu Ihrem
> und unserem Schutz die rechtliche Verbindlichkeit der vorstehenden
> Erklärungen ausschließen müssen. Diese Regelung gilt nur dann nicht,
> wenn wir mit Ihnen eine anderweitige schriftliche Vereinbarung über
> die Einhaltung von Sicherheits- und Verschlüsselungsstandards
> getroffen haben.
> 
> This e-mail contains confidential and/or privileged information. If
> you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail.
> Any unauthorised copying, disclosure or distribution of the material
> in this e-mail is strictly forbidden. As you know, the security of
> e-mail transmissions can not be guaranteed. E-mails can be misused
> to be written or modified under false names. For that reason, we ask
> you to understand the necessity for us to rule out the legal
> obligation of the above statement, for your protection and ours.
> This regulation is only invalid if we have concluded a special
> written agreement with you about the compliance with security and
> encryption standards.
> 
> REpower Systems SE · Sitz: Hamburg · Vorstand: Andreas Nauen
> (Vorsitz), Matthias Schubert, Marcus A. Wassenberg, Vinod R. Tanti ·
> Aufsichtsratsvorsitzender: Tulsi Tanti · Registergericht: AG Hamburg
> (Mitte) HRB Nr.: 118644
> 
> _______________________________________________
> Kst mailing list
> Kst at kde.org
> https://mail.kde.org/mailman/listinfo/kst
> 