[Kst] reading in NetCDF matrix (2-dim vector)

[Tom Downes]

I take back what I said on sync().  It seems to be important for the program
to recognize that the netCDF file is growing.  On the netCDF writing side, a
sync() command is equivalent to flushing an I/O buffer and this NC_SHARE
setting tries to buffer flush optimally and automatically.  I guess on the
reading side it's more like an update of the file and record sizes.

I thought that I had tested this on a growing file, but apparently not.  My
other suggestion for the Handbook stands.

Yours,

Tom

On Tue, May 11, 2010 at 1:56 PM, Tom Downes <tpdownes at caltech.edu> wrote:

Nicolas / Barth:
>
> I made a number of modifications to the netCDF reader.  The primary one is
> regarding the matrix problem I was having.  Read the comment below for
> details.
>
> The other potentially significant change I made is that I commented out a
> sync() command for the netCDF file.  My understanding is that, from a
> performance perspective, it is much better for the user to write to the NC
> file while enabling the NC_SHARE mode, which attempts to optimize syncing.
>  In fact, I think the sync command might be disabled if NC_SHARE is on.
>  Anyhow, I would suggest adding NC_SHARE to KST's documentation on reading
> netCDF.
>
> In a similar vein, I suggest editing the description of the update()
> datasource function on p. 203 of the KST handbook:
>
> "this function should read any new data..."
>
> Should instead read
>
> "this function should see if new data exists and update the frame count and
> other member variables appropriately..."
>
> update() does not actually read data so it should not be suggested to the
> user that it does.
>
> I'm not quite sure this code is ready for prime-time but if you have a
> chance to look at the changes, I would welcome comments.  In particular, I
> think I may have broken the INDEX field somehow.  I can't seem to create
> plots that have it as an x-axis vector, whereas everything else works just
> fine.
>
>   // TPD 10 May 2010 If the netCDF variable has 2 dimensions, then
>   // look to see if that variable has an attribute named "matrix"
>   // with a value of 1.  If so, treat the variable as a set of vectors
>   // rather than as a set of frames with multiple samples per frame
>   // (the ability to do this was introduced by K. Scott in her 2005
>   // revision).  This code shouldn't break the frame concept option.
>   //
>   // netCDF imposes the restriction that the UNLIMITED
>   // dimension (typically time) must come first in the dimension list
>   // so the second dimensions ends up being the number of rows.
>   //
>   // Subsequent functions (readField, update, etc) will recognize the
>   // fields as being from a matrix by the existence of a '/' character
>   // in the field name with the netCDF variable name ot the left and
>   // the row number .
>
> Tom
>
> On Thu, Apr 22, 2010 at 8:40 AM, Tom Downes <tpdownes at caltech.edu> wrote:
>
>> I write the data as a matrix because I receive the data as UDP packets of
>> 1xN_detectors every 10ms (100Hz sampling).  It seems more efficient to do 1
>> write-to-disk of N_detectors samples than N_detectors writes of 1 sample -
>> but I haven't actually tested this.  The existing analysis software assumes
>> a matrix so that is another reason to keep it that way.
>>
>> I am working in Matlab which doesn't provide for user-accessible
>> multi-threading and therefore makes building a set of FIFO buffers more
>> difficult.  Matlab so far has proven to be able to handle being a UDP client
>> without losing data.  I'd rather not have to build a whole UDP client in C
>> or C++ where dividing the 1xN_detectors samples into N_detectors FIFO
>> buffers would be easier.
>>
>> The only metadata I care about are scaling factors and offsets for
>> conversion to physical units, which are the same for all detectors.  I don't
>> think matrices are all that uncommon a way of doing things in NetCDF land -
>> half the point of NetCDF is to be able to expand the dimensions of your
>> matrices in a manner that is both efficient to store and to read.
>>
>> KST 2b2 wouldn't even recognize a NetCDF file for what it was and I
>> couldn't get any of the SVN checkouts to compile.  So I installed KDE3
>> alongside KDE4 - works just fine.  All you need are the kde3 libraries - you
>> can continue to login using KDE4, if that's your preference.
>>
>> I'll look at the datasource code and I'll be sure to share any work I do -
>> since the Matlab UDP client is working brilliantly, it may turn out to be
>> simpler to build a very stupid C program that just re-presents the data to
>> KST.
>>
>> Thanks for getting back,
>>
>> Tom
>>
>>
>> On Thu, Apr 22, 2010 at 7:17 AM, Brisset, Nicolas <
>> Nicolas.Brisset at eurocopter.com> wrote:
>>
>>>  Hi Tom,
>>>
>>>
>>>
>>> the netcdf datasource for kst currently does not support reading in
>>> matrices. It should however be fairly easy to add if you feel so inclined,
>>> knowing that matrices are less well supported than vectors as far as I know.
>>> I myself have only worked with vectors up to now, so I may not be aware of
>>> everything. But I fear there is no easy way to slice matrices into vectors
>>> (). Plus it makes little sense to me, as each variable may have metadata and
>>> such attached to it and you would lose all that by grouping vectors into a
>>> matrix. That said, I don't know your constraints well enough. Maybe you have
>>> a good reason indeed to do it that way.
>>>
>>> Another path you may want to go is making some changes to the netcdf
>>> datasource so that when it finds matrices it provides their lines and
>>> columns as extra vectors. In that case all the work should be done at
>>> datasource level, but that definitely sounds feasible. You could query the
>>> dimensions of your 2-dimensional variables and add to the field list vectors
>>> like Matrix1/Column1, Matrix1/Column2, Matrix1/Line1, etc…
>>>
>>> Note: in the current code, it seems that the dimensions of netcdf vars
>>> are not checked, that is **bad**. For cdf, it is done better.
>>>
>>>
>>>
>>> Then, the other question you may ask yourself is whether to use kst 1.9.x
>>> or 2.x for that work. Kst 2.x is still beta quality, but starting to work
>>> and there have been quite a few changes (among others wrt datasources) so
>>> that what you do for one version of the netcdf datasource may require
>>> substantial porting to the other one. FYI, kst2 also works under Windows but
>>> nobody has yet tried netcdf under Windows. So it is still definitely
>>> experimental.
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Nicolas
>>>
>>>
>>>
>>>  *EUROCOPTER DEUTSCHLAND GmbH*  *
>>> Sitz der Gesellschaft / Registered Office: Donauwörth*  *
>>> Registergericht / Registration Court: Amtsgericht Augsburg HRB 16508*  *
>>> Vorsitzender des Aufsichtsrates / Chairman of the Supervisory Board: Dr. Lutz Bertling
>>> *  *Geschäftsführung / Board of Management:*  *Dr. Wolfgang Schoder, Vorsitzender / CEO; Friedrich-Wilhelm Hormel; Ralf
>>> Barnscheidt*         CONFIDENTIALITY NOTICE
>>> This communication and the information it contains is intended for the addressee ( s ) named above and for no other persons or organizations.
>>> It is confidential and may be legally privileged and protected by law. The unauthorized use, copying or disclosure of this communication
>>> or any part of it is prohibited and may be unlawful.
>>> If you have received this communication in error, kindly notify us by return e-mail and discard and/or delete the communication.
>>> Thank you very much.
>>> It is possible for e-mails to be intercepted or affected by viruses. Whilst we maintain virus checks on our e-mails, we accept no liability for viruses
>>> or other material which might be introduced with this message.
>>> ------------------------------
>>>
>>> *Von:* Tom Downes [mailto:tpdownes at caltech.edu]
>>> *Gesendet:* Montag, 19. April 2010 20:46
>>> *An:* kst at kde.org
>>> *Betreff:* [Kst] reading in NetCDF matrix (2-dim vector)
>>>
>>>
>>>
>>> Hi:
>>>
>>>
>>>
>>> I'm reasonably familiar with KST but am having a lot of trouble parsing a
>>> NetCDF file in the fashion I intend.  I am working on an instrument that
>>> will have have around 100 independent detectors measured at 100 samples per
>>> second each (i.e., around 10,000 total measurements per second).  From the
>>> perspective of data acquisition it is much easier to write to a NetCDF
>>> variable with dimensions n_detectors x n_samples, where n_detectors would be
>>> this number around 100 (126 in example below) and n_samples would equal 100
>>> for 1 seconds worth of data.
>>>
>>>
>>>
>>> But trying to get KST to read this data in the way I intend seems to be
>>> difficult.  I want it to be able to split the matrix into 126 separate
>>> vectors so that I can plot a timestream for each - realistically I would
>>> take a representative sample.  The "sample" and "frame" terminology doesn't
>>> seem to quite match up to what I am doing - frames seem to be more about
>>> bookkeeping than actually combining many timestream vectors into a matrix.
>>>
>>>
>>>
>>> Anyhow, I would like to solve this in a relatively simple way, with a
>>> preference toward avoiding re-writing how the data is originally written to
>>> disk as it is much simpler this way.  Can I do what I need to do within KST
>>> or with a simple plug-in?  I am quite comfortable working in C/C++/Java and
>>> shell scripting environments.  Suggestions?
>>>
>>>
>>>
>>> This is what the file looks like:
>>>
>>>
>>>
>>> netcdf test-udp {
>>>
>>> dimensions:
>>>
>>>           time = UNLIMITED ; // (3960037 currently)
>>>
>>>           nkids = 126 ;
>>>
>>> variables:
>>>
>>>           int kids(time, nkids) ;
>>>
>>>                       kids:scale_factor = 1 ;
>>>
>>>                       kids:add_offset = 0 ;
>>>
>>>                       kids:units = "bits" ;
>>>
>>>           int timestamp(time) ;
>>>
>>>                       timestamp:scale_factor = 1 ;
>>>
>>>                       timestamp:add_offset = 0 ;
>>>
>>>                       timestamp:units = "bits" ;
>>>
>>> }
>>>
>>>
>>>
>>> I am running KST 1.9.1 compiled manually after checking out the release
>>> source.  I don't think it matters, but I am running the kde3 libraries
>>> alongside the kde4 libraries while logged into a KDE 3.5 WM session in
>>> OpenSUSE 11.2.
>>>
>>>
>>>
>>> Tom
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.kde.org/pipermail/kst/attachments/20100511/cadd7ba5/attachment-0001.htm