[Kst] reading in NetCDF matrix (2-dim vector)

Tue May 11 22:56:16 CEST 2010

Nicolas / Barth:

I made a number of modifications to the netCDF reader.  The primary one is
regarding the matrix problem I was having.  Read the comment below for
details.

The other potentially significant change I made is that I commented out a
sync() command for the netCDF file.  My understanding is that, from a
performance perspective, it is much better for the user to write to the NC
file while enabling the NC_SHARE mode, which attempts to optimize syncing.
 In fact, I think the sync command might be disabled if NC_SHARE is on.
 Anyhow, I would suggest adding NC_SHARE to KST's documentation on reading
netCDF.

In a similar vein, I suggest editing the description of the update()
datasource function on p. 203 of the KST handbook:

"this function should read any new data..."

Should instead read

"this function should see if new data exists and update the frame count and
other member variables appropriately..."

update() does not actually read data so it should not be suggested to the
user that it does.

I'm not quite sure this code is ready for prime-time but if you have a
chance to look at the changes, I would welcome comments.  In particular, I
think I may have broken the INDEX field somehow.  I can't seem to create
plots that have it as an x-axis vector, whereas everything else works just
fine.

  // TPD 10 May 2010 If the netCDF variable has 2 dimensions, then
  // look to see if that variable has an attribute named "matrix"
  // with a value of 1.  If so, treat the variable as a set of vectors
  // rather than as a set of frames with multiple samples per frame
  // (the ability to do this was introduced by K. Scott in her 2005
  // revision).  This code shouldn't break the frame concept option.
  //
  // netCDF imposes the restriction that the UNLIMITED
  // dimension (typically time) must come first in the dimension list
  // so the second dimensions ends up being the number of rows.
  //
  // Subsequent functions (readField, update, etc) will recognize the
  // fields as being from a matrix by the existence of a '/' character
  // in the field name with the netCDF variable name ot the left and
  // the row number .

Tom

On Thu, Apr 22, 2010 at 8:40 AM, Tom Downes <tpdownes at caltech.edu> wrote:

> I write the data as a matrix because I receive the data as UDP packets of
> 1xN_detectors every 10ms (100Hz sampling).  It seems more efficient to do 1
> write-to-disk of N_detectors samples than N_detectors writes of 1 sample -
> but I haven't actually tested this.  The existing analysis software assumes
> a matrix so that is another reason to keep it that way.
>
> I am working in Matlab which doesn't provide for user-accessible
> multi-threading and therefore makes building a set of FIFO buffers more
> difficult.  Matlab so far has proven to be able to handle being a UDP client
> without losing data.  I'd rather not have to build a whole UDP client in C
> or C++ where dividing the 1xN_detectors samples into N_detectors FIFO
> buffers would be easier.
>
> The only metadata I care about are scaling factors and offsets for
> conversion to physical units, which are the same for all detectors.  I don't
> think matrices are all that uncommon a way of doing things in NetCDF land -
> half the point of NetCDF is to be able to expand the dimensions of your
> matrices in a manner that is both efficient to store and to read.
>
> KST 2b2 wouldn't even recognize a NetCDF file for what it was and I
> couldn't get any of the SVN checkouts to compile.  So I installed KDE3
> alongside KDE4 - works just fine.  All you need are the kde3 libraries - you
> can continue to login using KDE4, if that's your preference.
>
> I'll look at the datasource code and I'll be sure to share any work I do -
> since the Matlab UDP client is working brilliantly, it may turn out to be
> simpler to build a very stupid C program that just re-presents the data to
> KST.
>
> Thanks for getting back,
>
> Tom
>
>
> On Thu, Apr 22, 2010 at 7:17 AM, Brisset, Nicolas <
> Nicolas.Brisset at eurocopter.com> wrote:
>
>>  Hi Tom,
>>
>>
>>
>> the netcdf datasource for kst currently does not support reading in
>> matrices. It should however be fairly easy to add if you feel so inclined,
>> knowing that matrices are less well supported than vectors as far as I know.
>> I myself have only worked with vectors up to now, so I may not be aware of
>> everything. But I fear there is no easy way to slice matrices into vectors
>> (). Plus it makes little sense to me, as each variable may have metadata and
>> such attached to it and you would lose all that by grouping vectors into a
>> matrix. That said, I don't know your constraints well enough. Maybe you have
>> a good reason indeed to do it that way.
>>
>> Another path you may want to go is making some changes to the netcdf
>> datasource so that when it finds matrices it provides their lines and
>> columns as extra vectors. In that case all the work should be done at
>> datasource level, but that definitely sounds feasible. You could query the
>> dimensions of your 2-dimensional variables and add to the field list vectors
>> like Matrix1/Column1, Matrix1/Column2, Matrix1/Line1, etc…
>>
>> Note: in the current code, it seems that the dimensions of netcdf vars are
>> not checked, that is **bad**. For cdf, it is done better.
>>
>>
>>
>> Then, the other question you may ask yourself is whether to use kst 1.9.x
>> or 2.x for that work. Kst 2.x is still beta quality, but starting to work
>> and there have been quite a few changes (among others wrt datasources) so
>> that what you do for one version of the netcdf datasource may require
>> substantial porting to the other one. FYI, kst2 also works under Windows but
>> nobody has yet tried netcdf under Windows. So it is still definitely
>> experimental.
>>
>>
>>
>> Regards,
>>
>>
>>
>> Nicolas
>>
>>
>>
>>  *EUROCOPTER DEUTSCHLAND GmbH*  *
>> Sitz der Gesellschaft / Registered Office: Donauwörth*  *
>> Registergericht / Registration Court: Amtsgericht Augsburg HRB 16508*  *
>> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Dr. Lutz Bertling
>> *  *Geschäftsführung / Board of Management:*  *Dr. Wolfgang Schoder, Vorsitzender / CEO; Friedrich-Wilhelm Hormel; Ralf
>> Barnscheidt*         CONFIDENTIALITY NOTICE
>> This communication and the information it contains is intended for the addressee ( s ) named above and for no other persons or organizations.
>> It is confidential and may be legally privileged and protected by law. The unauthorized use, copying or disclosure of this communication
>> or any part of it is prohibited and may be unlawful.
>> If you have received this communication in error, kindly notify us by return e-mail and discard and/or delete the communication.
>> Thank you very much.
>> It is possible for e-mails to be intercepted or affected by viruses. Whilst we maintain virus checks on our e-mails, we accept no liability for viruses
>> or other material which might be introduced with this message.
>> ------------------------------
>>
>> *Von:* Tom Downes [mailto:tpdownes at caltech.edu]
>> *Gesendet:* Montag, 19. April 2010 20:46
>> *An:* kst at kde.org
>> *Betreff:* [Kst] reading in NetCDF matrix (2-dim vector)
>>
>>
>>
>> Hi:
>>
>>
>>
>> I'm reasonably familiar with KST but am having a lot of trouble parsing a
>> NetCDF file in the fashion I intend.  I am working on an instrument that
>> will have have around 100 independent detectors measured at 100 samples per
>> second each (i.e., around 10,000 total measurements per second).  From the
>> perspective of data acquisition it is much easier to write to a NetCDF
>> variable with dimensions n_detectors x n_samples, where n_detectors would be
>> this number around 100 (126 in example below) and n_samples would equal 100
>> for 1 seconds worth of data.
>>
>>
>>
>> But trying to get KST to read this data in the way I intend seems to be
>> difficult.  I want it to be able to split the matrix into 126 separate
>> vectors so that I can plot a timestream for each - realistically I would
>> take a representative sample.  The "sample" and "frame" terminology doesn't
>> seem to quite match up to what I am doing - frames seem to be more about
>> bookkeeping than actually combining many timestream vectors into a matrix.
>>
>>
>>
>> Anyhow, I would like to solve this in a relatively simple way, with a
>> preference toward avoiding re-writing how the data is originally written to
>> disk as it is much simpler this way.  Can I do what I need to do within KST
>> or with a simple plug-in?  I am quite comfortable working in C/C++/Java and
>> shell scripting environments.  Suggestions?
>>
>>
>>
>> This is what the file looks like:
>>
>>
>>
>> netcdf test-udp {
>>
>> dimensions:
>>
>>           time = UNLIMITED ; // (3960037 currently)
>>
>>           nkids = 126 ;
>>
>> variables:
>>
>>           int kids(time, nkids) ;
>>
>>                       kids:scale_factor = 1 ;
>>
>>                       kids:add_offset = 0 ;
>>
>>                       kids:units = "bits" ;
>>
>>           int timestamp(time) ;
>>
>>                       timestamp:scale_factor = 1 ;
>>
>>                       timestamp:add_offset = 0 ;
>>
>>                       timestamp:units = "bits" ;
>>
>> }
>>
>>
>>
>> I am running KST 1.9.1 compiled manually after checking out the release
>> source.  I don't think it matters, but I am running the kde3 libraries
>> alongside the kde4 libraries while logged into a KDE 3.5 WM session in
>> OpenSUSE 11.2.
>>
>>
>>
>> Tom
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.kde.org/pipermail/kst/attachments/20100511/09d9b3f7/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: netcdf.cpp
Type: application/octet-stream
Size: 13744 bytes
Desc: not available
Url : http://mail.kde.org/pipermail/kst/attachments/20100511/09d9b3f7/attachment-0001.dll 